Actions
Bug #11556
open
occasional API server badness: 'incomplete response received from application'
Story points:
-
Release:
Release relationship:
Auto
Description
Clients sometimes get the following response from our API server:
$ curl -H 'Authorization: OAuth2 '$ARVADOS_API_TOKEN https://$ARVADOS_API_HOST/arvados/v1/pipeline_instances/z8ta6-d1hrv-1gktmvpt7qmbsdi <h2>Incomplete response received from application</h2>
When this happens, it seems to happen a lot - i.e. it won't happen for a long while and then it will suddenly start happen with some frequency (usually once every few attempts, but sometimes for every attempt for a few minutes).
The only log entry in the API server's production log that seems relevant during this period is the following:
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds (waited 5.000092554 seconds). The max pool size is currently 5; consider increasing it.): app/middlewares/arvados_api_token.rb:59:in `call'
But it does not seem to happen on every request, it is just logged sometimes during the bad times.
I've now set `pool: 40` in database.yml so I'll see if that fixes anything. The postgresql logs during this time don't log any errors or any failed connection attempts, although the time I looked at was during a database checkpoint and vacuum operation. Maybe that has something to do with it?
The following are longer logs from a particular time when it was all going wrong (in which I was testing directly with curl but also workbench).
nginx frontend log:
172.17.180.10 - - [26/Apr/2017:17:10:05 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-q6hrrxo5meyzizi HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:06 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:06 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-xddgoh9x6i0h5x2 HTTP/1.1" 200 1469 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:06 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2889 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:06 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2889 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-wj93ifb7e2mh5rv HTTP/1.1" 200 1467 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2790 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-jx5akegebjn8ura HTTP/1.1" 200 1560 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2980 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2999 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:08 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:08 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-yggtg6lhlpck95w HTTP/1.1" 200 1466 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:09 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:09 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-obvxyuclj1qe2r5 HTTP/1.1" 200 1467 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:09 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:09 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-ghmssff5543ufib HTTP/1.1" 200 1469 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 200 2891 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-nbchnecnzposa6g HTTP/1.1" 200 1470 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 200 2891 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.125.15 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1gktmvpt7qmbsdi HTTP/1.1" 502 54 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 172.17.180.10 - - [26/Apr/2017:17:10:15 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:21 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ry2w3vzgm782jo3 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:27 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-e263otp5ar8wzpz HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:32 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ph1smfl4dzmkvhi HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:38 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-s30zwbvsqh1qe8z HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:43 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-vg672cm79xazoc5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:49 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xkwkeil2pobdybv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:10:54 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-355ug7lekye1phh HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:00 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-jmmz0393gb8mdr5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:06 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tknxo8hy3iuzgog HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:11 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sy5bwcgxdurz6d8 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:17 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-g7c03mnyzpr68d0 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:22 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-zj9ppkcmko7r29h HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:28 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-pek97c6v654lrpr HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:34 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xqg4sunuvoi72ut HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:39 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-7t1tql10quanr3l HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:45 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-wg99e7frx8wdnjz HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:51 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-naihasmk4hdlik1 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:11:56 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-y3gxksgzrxvow2m HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:02 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-a9tb0piltvoey2p HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-wsrigv4a4lk0qs6 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:13 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-fg4glvdnxmavw9e HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:18 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-q6xqybumiyunarv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:24 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-y8slip5m13weody HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:30 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-eywn1ihktck4wmp HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:35 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-czx55wpgormgz70 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:41 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-k8se442ey5cs2oe HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:46 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-cc07ckhohihnr2r HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:52 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xphz9nbp1mmt3u9 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:12:57 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-9ss82yz9udcmes2 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:03 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-s1001dwbwqqlmbv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:09 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-td9my0b3itehjyb HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:14 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1sphdcriaumh0nx HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:20 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-zd8617jr0z2cxlg HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:25 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-8ju3etb13302xqn HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:31 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-621s20xny8bxikk HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:37 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1nyteeyqb98nun5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:42 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sln2pccinu28wtj HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:48 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-2chtr2pnzj0k8et HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:53 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-7y9eduhe4kudhz6 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:13:59 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-hqrkwgif0tiyxzq HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 172.17.180.10 - - [26/Apr/2017:17:14:04 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-cdcxtcq4vu3jfnk HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic"
nginx backend log:
127.0.0.1 - - [26/Apr/2017:17:10:05 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-q6hrrxo5meyzizi HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:06 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:06 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-xddgoh9x6i0h5x2 HTTP/1.1" 200 1469 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:06 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2889 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:06 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 HTTP/1.1" 200 2889 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-wj93ifb7e2mh5rv HTTP/1.1" 200 1467 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2790 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-jx5akegebjn8ura HTTP/1.1" 200 1560 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:07 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2980 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz HTTP/1.1" 200 2999 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:08 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:08 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-yggtg6lhlpck95w HTTP/1.1" 200 1466 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:08 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf HTTP/1.1" 200 2886 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:09 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:09 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-obvxyuclj1qe2r5 HTTP/1.1" 200 1467 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:09 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:09 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 HTTP/1.1" 200 2888 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-ghmssff5543ufib HTTP/1.1" 200 1469 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya HTTP/1.1" 200 2890 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 200 2891 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/jobs/z8ta6-8i9sb-nbchnecnzposa6g HTTP/1.1" 200 1470 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 200 2891 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1gktmvpt7qmbsdi HTTP/1.1" 502 54 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 127.0.0.1 - - [26/Apr/2017:17:10:15 +0100] "PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:21 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ry2w3vzgm782jo3 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:27 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-e263otp5ar8wzpz HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:32 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ph1smfl4dzmkvhi HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:38 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-s30zwbvsqh1qe8z HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:43 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-vg672cm79xazoc5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:49 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xkwkeil2pobdybv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:10:54 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-355ug7lekye1phh HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:00 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-jmmz0393gb8mdr5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:06 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tknxo8hy3iuzgog HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:11 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sy5bwcgxdurz6d8 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:17 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-g7c03mnyzpr68d0 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:22 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-zj9ppkcmko7r29h HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:28 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-pek97c6v654lrpr HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:34 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xqg4sunuvoi72ut HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:39 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-7t1tql10quanr3l HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:45 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-wg99e7frx8wdnjz HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:51 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-naihasmk4hdlik1 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:11:56 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-y3gxksgzrxvow2m HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:02 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-a9tb0piltvoey2p HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:07 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-wsrigv4a4lk0qs6 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:13 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-fg4glvdnxmavw9e HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:18 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-q6xqybumiyunarv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:24 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-y8slip5m13weody HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:30 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-eywn1ihktck4wmp HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:35 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-czx55wpgormgz70 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:41 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-k8se442ey5cs2oe HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:46 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-cc07ckhohihnr2r HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:52 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-xphz9nbp1mmt3u9 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:12:57 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-9ss82yz9udcmes2 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:03 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-s1001dwbwqqlmbv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:09 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-td9my0b3itehjyb HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:14 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1sphdcriaumh0nx HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:20 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-zd8617jr0z2cxlg HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:25 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-8ju3etb13302xqn HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:31 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-621s20xny8bxikk HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:37 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1nyteeyqb98nun5 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:42 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sln2pccinu28wtj HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:48 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-2chtr2pnzj0k8et HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:53 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-7y9eduhe4kudhz6 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:13:59 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-hqrkwgif0tiyxzq HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:04 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-cdcxtcq4vu3jfnk HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:10 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-i76jzgpmapgynfp HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:16 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-zs6t7q7qaj7u68y HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:21 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-0q7vnr1jk7llqcv HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:27 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3ap7uujvcj9mfzo HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:32 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sijykyj14ou92xp HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:38 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-4o0k30x6j6i2i3q HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:43 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ao02rc2xx6zkm8m HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:49 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-uopo4j8k5qnc8qi HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:14:55 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-p6eog7u2ifgk1x8 HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic" 127.0.0.1 - - [26/Apr/2017:17:15:00 +0100] "GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-sv2vl2pr4dx3blq HTTP/1.1" 502 54 "-" "arv-run-pipeline-instance/0.0 google-api-ruby-client/0.8.7 Linux/3.13.0-88-generic"
api server production.log:
[api.arvados.sanger] [4da2daffbfedb312509d456373ca6773] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":5.81,"view":0.49,"db":0.54,"params":{},"@timestamp":"2017-04-26T16:10:06.446231513Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [2c4efc6f6d56ae96adae5a9d932ee8a2] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-xddgoh9x6i0h5x2","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.41,"view":0.27,"db":1
.26,"params":{},"@timestamp":"2017-04-26T16:10:06.472321094Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-xddgoh9x6i0h5x2 (arvados/v1/jobs#show)"}
[api.arvados.sanger] [75c1098d9a9c28638aeb3003bbeb2626] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [75c1098d9a9c28638aeb3003bbeb2626] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":30.98,"view":0.18,"db":19.88,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\
"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"f9c6d
635baa8fb22c9cbd2821afcc838+958\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"mi
n_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-xddgoh9x6i0h5x2\",\"kind\":\"arvados#job\",\"etag\":\"6cl04enqsqhgqfxeuudqir5eb\",\"uuid\":\"z8ta6-8i9sb-xddgoh9x6i0h5x2\",\"o
wner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T10:46:00.129856000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-17T19:57:43.217245000Z\"
,\"submit_id\":\"instance z8ta6-d1hrv-5xq1r8b5b8wmg72 rand 2csjwuhm2ttm8\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_co
llection\":\"f9c6d635baa8fb22c9cbd2821afcc838+958\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_u
uid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-17T18:48:26.157980000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2tw
vc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":154,\"running\":9,\"failed\":0,\"todo\":38},\"nonde
terministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator[...]","@timestamp":"2017-04-26T16:10:06.520360152Z","@version":"1","message":"[200] PUT /arvados/
v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [4f7c8b10678dc4d8594a2ec9918479a8] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [4f7c8b10678dc4d8594a2ec9918479a8] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-5xq1r8b5b8wmg72","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":37.35,"view":0.22,"db":26.03,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:10:06.576907929Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z
8ta6-d1hrv-5xq1r8b5b8wmg72 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [2930eeefaf80c9a540ab736d5953209e] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":4.04,"view":0.24,"db":0.46,"params":{},"@timestamp":"2017-04-26T16:10:07.152860997Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [be5970c6fbf4fffaa3641ab85c4d5b00] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-wj93ifb7e2mh5rv","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.36,"view":0.34,"db":1
.1,"params":{},"@timestamp":"2017-04-26T16:10:07.179384745Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-wj93ifb7e2mh5rv (arvados/v1/jobs#show)"}
[api.arvados.sanger] [9a7c1ce07790a93ceb596d780caaf1b8] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [9a7c1ce07790a93ceb596d780caaf1b8] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":30.27,"view":0.2,"db":20.07,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"
script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"61cc29
88deb9d2b49c519a244dec6591+1084\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"mi
n_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-wj93ifb7e2mh5rv\",\"kind\":\"arvados#job\",\"etag\":\"eqx5b7ybmb6c7gcq6h0z0d90m\",\"uuid\":\"z8ta6-8i9sb-wj93ifb7e2mh5rv\",\"o
wner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:08:06.013918000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T02:15:50.624406000Z\"
,\"submit_id\":\"instance z8ta6-d1hrv-1mnay4ehpite15z rand 17vw14we2cz4l\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_co
llection\":\"61cc2988deb9d2b49c519a244dec6591+1084\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_
uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T02:15:44.468913000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2t
wvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":1,\"failed\":0,\"todo\":0},\"nondete
rministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\"[...]","@timestamp":"2017-04-26T16:10:07.227576962Z","@version":"1","message":"[200] PUT /arvados/
v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [c78c637f69b320c0a2271539cf848fe4] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [c78c637f69b320c0a2271539cf848fe4] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-1mnay4ehpite15z","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":47.86,"view":0.16,"db":37.85,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:10:07.292140185Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z
8ta6-d1hrv-1mnay4ehpite15z (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [416baa70a2beb6edeec05978c6e0d15d] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":4.18,"view":0.2,"db":0.57,"params":{},"@timestamp":"2017-04-26T16:10:07.879944849Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [b3817ddfcce7af268a60d6e525c910c4] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-jx5akegebjn8ura","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.29,"view":0.26,"db":0
.99,"params":{},"@timestamp":"2017-04-26T16:10:07.904231955Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-jx5akegebjn8ura (arvados/v1/jobs#show)"}
[api.arvados.sanger] [20a79a7fee69b39aea800f3a52e83940] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [20a79a7fee69b39aea800f3a52e83940] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":15.21,"view":0.23,"db":4.25,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"
script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"80ea92
627e393e659567dc02b5623fb5+958\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min
_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-jx5akegebjn8ura\",\"kind\":\"arvados#job\",\"etag\":\"952xcsx3gqlvplepvib08erzu\",\"uuid\":\"z8ta6-8i9sb-jx5akegebjn8ura\",\"ow
ner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-26T02:21:46.120077000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-26T12:07:47.493644000Z\",
\"submit_id\":\"instance z8ta6-d1hrv-tkxeem3epv5ovzz rand 2bg9nesh85dlx\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_col
lection\":\"80ea92627e393e659567dc02b5623fb5+958\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uu
id\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-26T12:07:18.640768000Z\",\"finished_at\":\"2017-04-26T12:07:47.493644000Z\",\"output\":\"d41d8cd98f00b204e9800998ecf8427e+0\",\"success\":false,\"running\":false,\"stat
e\":\"Failed\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":\"3a19f54daaf52a3a7b01d5e5535daef3+85\",\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_
mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":1,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr[...]","@timestamp":"2017-04-26T16:10:07.986799722Z","@version":"1","message":"[200] PUT /arvados/v1/
pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [6eecbd095b1584decd7c03251a3687ab] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [6eecbd095b1584decd7c03251a3687ab] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":29.61,"view":0.15,"db":19.48,"params":{"pipeline_instance":{"state":"Failed","finished_at":"2017-04-26 17:10:07 +0100","components_summary":{"todo":0,"done":0,"failed":1}}},"@timestamp":"2017-04-26T16:10:08.034525034Z","@version
":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tkxeem3epv5ovzz (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [14034581d854259e90fee0edfc2032cf] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":4.05,"view":0.19,"db":0.54,"params":{},"@timestamp":"2017-04-26T16:10:08.615890145Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [7a6721301c4fa772c13cc5c1b2266927] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-yggtg6lhlpck95w","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.23,"view":0.21,"db":1
.02,"params":{},"@timestamp":"2017-04-26T16:10:08.638970334Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-yggtg6lhlpck95w (arvados/v1/jobs#show)"}
[api.arvados.sanger] [3c108617baaceab58af25d72ba70c079] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [3c108617baaceab58af25d72ba70c079] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":28.78,"view":0.21,"db":18.91,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\
"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"7e83d
4ae661afd4dccadb6dae79b7892+916\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"mi
n_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-yggtg6lhlpck95w\",\"kind\":\"arvados#job\",\"etag\":\"8yfd472ef3j8f0rsoomfb6e59\",\"uuid\":\"z8ta6-8i9sb-yggtg6lhlpck95w\",\"o
wner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:21:05.836141000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:39:08.545447000Z\"
,\"submit_id\":\"instance z8ta6-d1hrv-3yjh7749twxrqcf rand 3gt98fey7g9o8\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_co
llection\":\"7e83d4ae661afd4dccadb6dae79b7892+916\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_u
uid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:39:07.652346000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2tw
vc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeter
ministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\":\"[...]","@timestamp":"2017-04-26T16:10:08.684046415Z","@version":"1","message":"[200] PUT /arvado
s/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [3e6abab5e03f22e4c053629fd39b7fbc] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [3e6abab5e03f22e4c053629fd39b7fbc] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3yjh7749twxrqcf","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":57.54,"view":0.23,"db":46.45,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:10:08.760490811Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z
8ta6-d1hrv-3yjh7749twxrqcf (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [d488cf156f78f190f0659941ce5493cf] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":3.99,"view":0.24,"db":0.46,"params":{},"@timestamp":"2017-04-26T16:10:09.365358606Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [ea2e8ff48c459c8ee35edf8f7a2268e0] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-obvxyuclj1qe2r5","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.18,"view":0.19,"db":0
.93,"params":{},"@timestamp":"2017-04-26T16:10:09.388070978Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-obvxyuclj1qe2r5 (arvados/v1/jobs#show)"}
[api.arvados.sanger] [fbcccb971f28a737acd80369d435720d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [fbcccb971f28a737acd80369d435720d] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":41.41,"view":0.22,"db":31.52,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"501eca4e794ef9854ee85a40ab5681d7+1000\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-obvxyuclj1qe2r5\",\"kind\":\"arvados#job\",\"etag\":\"50u7j2ghtjiwlbpvki8p5d7ka\",\"uuid\":\"z8ta6-8i9sb-obvxyuclj1qe2r5\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:22:08.816979000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:44:23.306392000Z\",\"submit_id\":\"instance z8ta6-d1hrv-oaj6wg7xb1azxq7 rand 3umtqxlqt54ml\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"501eca4e794ef9854ee85a40ab5681d7+1000\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:44:22.560425000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\"[...]","@timestamp":"2017-04-26T16:10:09.445645139Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [60f5c280c4a407f92d4ca3a9eff15897] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [60f5c280c4a407f92d4ca3a9eff15897] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":29.63,"view":0.16,"db":19.87,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:10:09.491481418Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-oaj6wg7xb1azxq7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [0564212081913c0f9393f5509ecd6f5d] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":9.55,"view":0.55,"db":0.82,"params":{},"@timestamp":"2017-04-26T16:10:10.085022819Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [97a971e3c92f9b73f4482f30813f89c7] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-ghmssff5543ufib","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":22.84,"view":0.43,"db":1.45,"params":{},"@timestamp":"2017-04-26T16:10:10.133074194Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-ghmssff5543ufib (arvados/v1/jobs#show)"}
[api.arvados.sanger] [436cf040d92404decfe9ac91c00daab1] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [436cf040d92404decfe9ac91c00daab1] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":39.56,"view":0.26,"db":28.13,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"fa4b39f87cb34ddd183eb1fd59626d98+1168\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-ghmssff5543ufib\",\"kind\":\"arvados#job\",\"etag\":\"7czyl3rhnvypulk2i1va84kzl\",\"uuid\":\"z8ta6-8i9sb-ghmssff5543ufib\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-19T00:43:38.602755000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-19T04:14:40.653760000Z\",\"submit_id\":\"instance z8ta6-d1hrv-3c733sowg7wtbya rand 2sxtrm8zz1b09\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"fa4b39f87cb34ddd183eb1fd59626d98+1168\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-19T03:21:50.698265000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":200,\"running\":1,\"failed\":0,\"todo\":0},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locato[...]","@timestamp":"2017-04-26T16:10:10.193537360Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [5c39ac730687c9da46bdc49200dae45d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [5c39ac730687c9da46bdc49200dae45d] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":12.87,"view":0.16,"db":3.29,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:10:10.223768485Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-3c733sowg7wtbya (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [fee6549c1b4b824e17e0f1c4f5092d12] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":3.98,"view":0.19,"db":0.49,"params":{},"@timestamp":"2017-04-26T16:10:10.799677443Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [c9b6fe946d95e994483b5e6e2536e746] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-nbchnecnzposa6g","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.32,"view":0.22,"db":1.0,"params":{},"@timestamp":"2017-04-26T16:10:10.823130273Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-nbchnecnzposa6g (arvados/v1/jobs#show)"}
[api.arvados.sanger] [9d6d2f1e8f33493e40e65c182c502d39] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [9d6d2f1e8f33493e40e65c182c502d39] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":44.64,"view":0.19,"db":34.92,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"6b66c97bca57923a302c9187741fef71+1126\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-nbchnecnzposa6g\",\"kind\":\"arvados#job\",\"etag\":\"7q1r2n315yuy6o078wbui6kou\",\"uuid\":\"z8ta6-8i9sb-nbchnecnzposa6g\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:06:01.838973000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T02:17:52.925562000Z\",\"submit_id\":\"instance z8ta6-d1hrv-uh94fxofs6mvqti rand 362sl39e3ptas\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"6b66c97bca57923a302c9187741fef71+1126\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T01:15:44.362001000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":157,\"running\":1,\"failed\":0,\"todo\":43},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locat[...]","@timestamp":"2017-04-26T16:10:10.884698206Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-uh94fxofs6mvqti (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [ff05a0baa661545e21aca52780a0d020] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [fb6acdcbd5470901c3a7e8bb1a125270] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [ff05a0baa661545e21aca52780a0d020] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-1ckqm6rgqizti7z/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":28.94,"view":0.62,"db":14.0,"params":{"ping_secret":"ztxt3599ctdovhw6okc89zjya6qf6d78spt4jvmmu4u62nyhx","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.116725072Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-1ckqm6rgqizti7z/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [fb6acdcbd5470901c3a7e8bb1a125270] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-8thrz8n7g55nxnw/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":28.63,"view":0.87,"db":12.8,"params":{"ping_secret":"2ucyqohee5tjttajfn9uqvt5tsxzdaau0qb74bgmcvxzir4hml","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.121518232Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-8thrz8n7g55nxnw/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [9ca40d9a51bf859298563e797990857a] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.54,"view":0.32,"db":0.49,"params":{},"@timestamp":"2017-04-26T16:15:01.199481358Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [1ca2b6fd53413858b2f39ac0472cad0b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a1d73676493aad43b0ae2584c6e54a9a] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-xnurmhqhwi4uf3d","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.52,"view":0.35,"db":1.22,"params":{},"@timestamp":"2017-04-26T16:15:01.228469648Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-xnurmhqhwi4uf3d (arvados/v1/jobs#show)"}
[api.arvados.sanger] [22f8bf4504bbc73c883336b9e2796ecf] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [1ca2b6fd53413858b2f39ac0472cad0b] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-goq0yfgpwmoix02/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":38.34,"view":0.37,"db":30.05,"params":{"ping_secret":"nj23u2db7f15zbny1jj16eonvwdiyv3d2jpvubdw784d2cfg3","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.249849465Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-goq0yfgpwmoix02/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [e4c1ceeb41976265348093ed1cb0e674] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [73a3c6399762b00319059c8b84cd7caa] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [22f8bf4504bbc73c883336b9e2796ecf] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-onpouqgv8tr4yww/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":38.49,"view":0.36,"db":29.17,"params":{"ping_secret":"4ytreob90yvnvjldwxol7ndcr3knkz740u39n93vri43fb9tww","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.274471905Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-onpouqgv8tr4yww/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [7961ecba98cebaee55b8939f156e5697] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e4c1ceeb41976265348093ed1cb0e674] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-87puy14gjzvr3vh/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":37.51,"view":0.24,"db":29.86,"params":{"ping_secret":"44jsn9c7nuf4i6c620r5dus1y0e93bzsh9gp81r1w1lahxjyuv","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.290488464Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-87puy14gjzvr3vh/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [73a3c6399762b00319059c8b84cd7caa] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":61.04,"view":0.49,"db":47.79,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"523e08f59af1adb495594bc00290c629+1000\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-xnurmhqhwi4uf3d\",\"kind\":\"arvados#job\",\"etag\":\"clhkgzlk522dz4f8owxt2jh3g\",\"uuid\":\"z8ta6-8i9sb-xnurmhqhwi4uf3d\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-18T14:12:22.686107000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T21:53:16.910824000Z\",\"submit_id\":\"instance z8ta6-d1hrv-7v1rbrrmebpm3mp rand 1xyttwurzctlm\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"523e08f59af1adb495594bc00290c629+1000\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T21:09:04.739439000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":165,\"running\":7,\"failed\":0,\"todo\":29},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locat[...]","@timestamp":"2017-04-26T16:15:01.317064527Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [7961ecba98cebaee55b8939f156e5697] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-0lrjzbq96r4dxc2/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":27.19,"view":0.22,"db":19.04,"params":{"ping_secret":"40l0wzytl8wkw1ftsgsyid9vbxbmqpwtkn5mj8ocic4w0uo6u8","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.317986560Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-0lrjzbq96r4dxc2/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [1300be0afa2e7ea5bd12441c0963c24c] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [0c0c23f63d59cc4344b7b56495ad32e3] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d596bdc509d12a84c7ee66257210af91] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [0c0c23f63d59cc4344b7b56495ad32e3] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-eakdzltjtr3h2w0/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":17.47,"view":0.2,"db":10.17,"params":{"ping_secret":"4axvt7vaqhqot96geq08n37r76ijge0eujgezgpvwmvgkif533","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.358065402Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-eakdzltjtr3h2w0/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [1300be0afa2e7ea5bd12441c0963c24c] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":27.79,"view":0.28,"db":15.76,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:01.364427870Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-7v1rbrrmebpm3mp (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [b42e100ca4f1c5cb7d6a1ba06c579390] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e355b0989a1b69ded9d211d047a7ab7a] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [b42e100ca4f1c5cb7d6a1ba06c579390] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-h2jiyjx5a5d437p/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":23.94,"view":0.37,"db":14.13,"params":{"ping_secret":"603jdphqf44qr4z7znu81ub1aki2t7m3s1vaus3roidafups2q","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.397051790Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-h2jiyjx5a5d437p/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [b3687054bb64c4718c4d0bdd9e452c87] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d596bdc509d12a84c7ee66257210af91] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-acluu7h95fcy0ae/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":47.32,"view":0.
31,"db":39.2,"params":{"ping_secret":"e879l5tpkb1c1k71inet02z31xfe9szen0ovu0yc9csxn2u5q","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.401384774Z","@version":"1","message"
:"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-acluu7h95fcy0ae/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [e355b0989a1b69ded9d211d047a7ab7a] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-1i1v5zotflg26jn/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":21.53,"view":0.2,"db":13.78,"params":{"ping_secret":"2qt7o43l6av3vf514765sscjka2g0rkchx8vg3l8qs3wbhcgpc","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.412957466Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-1i1v5zotflg26jn/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6939815fa127369aef13ee105481eb9c] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [b3687054bb64c4718c4d0bdd9e452c87] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-2ybf3w0s1lxvvyl/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":39.96,"view":0.24,"db":31.99,"params":{"ping_secret":"5wdjor8v4hd3ib2dmm4sxbjtqwf7he2o7yd3tulngsgjo5zly1","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.440018194Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-2ybf3w0s1lxvvyl/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6939815fa127369aef13ee105481eb9c] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-a3o8ihbgt1cw1ng/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":33.59,"view":0.17,"db":15.6,"params":{"ping_secret":"4xa6ydqzmxgherk1gehepv9s4rdrlymuhjmv3c5t6vz3va2v6d","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.466903748Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-a3o8ihbgt1cw1ng/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [8a2c7de9014df977e775f46c9f25ef75] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [2f3ddb752260a38cf1b7918b29e5b8b5] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e7737725701cf57f13a2da10f4fcd2b8] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [8a2c7de9014df977e775f46c9f25ef75] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-su5w0k705lpb6v3/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":20.69,"view":0.23,"db":12.49,"params":{"ping_secret":"tejrdi5nqh3ajrx8e3iumw6dg9ox53ngw0xniypfbyq31nqx1","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.530786763Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-su5w0k705lpb6v3/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [2f3ddb752260a38cf1b7918b29e5b8b5] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ch62ym4wo4gh0xm/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":44.53,"view":0.21,"db":37.17,"params":{"ping_secret":"3s9pr2upha1t3kbwdq6n0gdf9aanl6ymoq8htijxgfwkly9mya","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.559025870Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ch62ym4wo4gh0xm/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [88924625a6d4bc36ddaf8e692d93dff1] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e7737725701cf57f13a2da10f4fcd2b8] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":42.15,"view":0.17,"db":34.17,"params":{"ping_secret":"19acih9yt5owqkku6afjg8v5166jnrvy717076afgzlpy5xc20","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.567890621Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [7ad8ef28a3c41fcdf1b2a6e2700687f4] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a966d031f232fb10a70c649cf41d0b58] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [88924625a6d4bc36ddaf8e692d93dff1] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-dcjuzp0fzjcceb1/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":49.45,"view":0.23,"db":28.8,"params":{"ping_secret":"3yk403toc4iaq70l5hs5qzb4ug011hkcwpvrqvptofg04rs5q6","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.610057879Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-dcjuzp0fzjcceb1/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [7ad8ef28a3c41fcdf1b2a6e2700687f4] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ztb4ykmctl8bt52/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":40.46,"view":0.29,"db":31.75,"params":{"ping_secret":"h4f9lc7utwx1scwl969cawg7yuh506cxtmd2moa6vv4klygyj","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.618176055Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ztb4ykmctl8bt52/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6246da2b34af85623a53306ba37854cb] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a966d031f232fb10a70c649cf41d0b58] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-kjcae6e2x2ze9wy/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":48.39,"view":0.29,"db":39.32,"params":{"ping_secret":"qndwiq4ny11xix5lz2m0mx7mfuyngmdvf7iiakxp6wq3ernyf","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.640835192Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-kjcae6e2x2ze9wy/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6246da2b34af85623a53306ba37854cb] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-sa1q59632vhxov6/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":46.12,"view":0.26,"db":38.17,"params":{"ping_secret":"297ksswrlqaq19pob6e0zndgs4bxs4qn162o01dizzgtolaezg","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.676077021Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-sa1q59632vhxov6/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [09db560425234abeca48aa1311cf76b0] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [09db560425234abeca48aa1311cf76b0] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-dw0j4kqtbn9b4xo/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":33.36,"view":0.16,"db":26.36,"params":{"ping_secret":"1n05zp5gwfq0gnqsu402wp7bgat18uf4eah03fha2zwmlw7rim","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.720786196Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-dw0j4kqtbn9b4xo/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [674c6291c339c684bf7e278e87cf9604] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [674c6291c339c684bf7e278e87cf9604] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-5qd5cdeiza4v6lv/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":19.08,"view":0.25,"db":11.16,"params":{"ping_secret":"692ib9l80dem8lz89rwtroi0pawae8xg7k1g63i9hz7ciiyqup","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.814936312Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-5qd5cdeiza4v6lv/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [08dadc638808706aa6bbaa8662b8a639] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [4fc3937a96576ac34f7df0a807f1dbfa] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [ba46efa451c7cd747304a347e25f781d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [08dadc638808706aa6bbaa8662b8a639] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-5co41k3w77aokje/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":28.09,"view":0.13,"db":21.36,"params":{"ping_secret":"453pd282knmh8nwohnehfhfr09ii9qkn9lp9ux0flxorrzjrbr","total_cpu_cores":"32","total_ram_mb":"257851","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.854237476Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-5co41k3w77aokje/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [e76180a68d7f033cb1e94a086a222c9c] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [4fc3937a96576ac34f7df0a807f1dbfa] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ekzlxvozts92sqm/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":27.58,"view":0.
16,"db":20.97,"params":{"ping_secret":"5tapdd6uj8509a5whjt4kshfa1jfrno1byd637nwwdj7nge88v","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.862102225Z","@version":"1","mess
age":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ekzlxvozts92sqm/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [ba46efa451c7cd747304a347e25f781d] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-avqsjfg0naek3sw/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":15.96,"view":0.21,"db":8.87,"params":{"ping_secret":"4zjbrw6xmfbltrm8k8ds3rzlbi4ezpze5n16ce2d7b7sldpl29","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:15:01.865080609Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-avqsjfg0naek3sw/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [e76180a68d7f033cb1e94a086a222c9c] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-y79anz2iox0q6j4/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":27.06,"view":0.22,"db":19.43,"params":{"ping_secret":"1m9lossetcrt1v2xotm8vsgiu1bcw23e110w4rd2x6xk0j44g4","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:01.886146627Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-y79anz2iox0q6j4/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [4cc73f77e616b7056efc610737604a15] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [301d72128a3c3018e7df9c7fb8353784] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.28,"view":0.33,"db":0.53,"params":{},"@timestamp":"2017-04-26T16:15:01.979822852Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [38396078d183acc97735ad9676687b85] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-acm531ute1kx6oj","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.41,"view":0.28,"db":1.06,"params":{},"@timestamp":"2017-04-26T16:15:02.004210752Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-acm531ute1kx6oj (arvados/v1/jobs#show)"}
[api.arvados.sanger] [4cc73f77e616b7056efc610737604a15] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-kot4vuz4fwa1wj6/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":46.25,"view":0.22,"db":38.61,"params":{"ping_secret":"68hbrge4twjshys8pndhnufryvobhgzek1c6oil0bqdv6svk5c","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:15:02.015149240Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-kot4vuz4fwa1wj6/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [d4487c5d7797a556fd921be6bff64329] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d4487c5d7797a556fd921be6bff64329] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":25.59,"view":0.32,"db":14.53,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"e2c8aedc61dcc4069cadf9f4b5f66f0a+1588\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-acm531ute1kx6oj\",\"kind\":\"arvados#job\",\"etag\":\"7xjtzw7xg0dtir94mbvbzyqt9\",\"uuid\":\"z8ta6-8i9sb-acm531ute1kx6oj\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-18T14:12:13.837267000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T21:53:26.847051000Z\",\"submit_id\":\"instance z8ta6-d1hrv-an73f53u6qhvllw rand 1j6wd4027oilj\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"e2c8aedc61dcc4069cadf9f4b5f66f0a+1588\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T21:04:57.944125000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":200,\"running\":1,\"failed\":0,\"todo\":0},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locato[...]","@timestamp":"2017-04-26T16:15:02.048030582Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [faf8c9ff1dac6328ab6827aa9f2620be] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [faf8c9ff1dac6328ab6827aa9f2620be] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":40.24,"view":0.22,"db":30.17,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:02.107789344Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-an73f53u6qhvllw (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [59eaddad00d8a167cf11b71ea63523c4] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.07,"view":0.26,"db":0.53,"params":{},"@timestamp":"2017-04-26T16:15:02.687826798Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [6fef0b9ee32c18effbabc4e080932a7c] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-plqyfahmo22bnzq","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.0,"view":0.24,"db":0.95,"params":{},"@timestamp":"2017-04-26T16:15:02.711921927Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-plqyfahmo22bnzq (arvados/v1/jobs#show)"}
[api.arvados.sanger] [94a327fb5c212dc4075aa528f66d971f] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [94a327fb5c212dc4075aa528f66d971f] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":14.06,"view":0.2,"db":3.78,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"f9dfec7d0b29613ef0d56cee6d1faccd+1168\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-plqyfahmo22bnzq\",\"kind\":\"arvados#job\",\"etag\":\"68wndzv52yx69dhq6ypbhe83e\",\"uuid\":\"z8ta6-8i9sb-plqyfahmo22bnzq\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-25T22:11:07.792616000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-25T22:51:54.426808000Z\",\"submit_id\":\"instance z8ta6-d1hrv-6h85anwkt785p2d rand 131brbk7zxk1h\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"f9dfec7d0b29613ef0d56cee6d1faccd+1168\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-25T22:11:09.800097000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":133,\"running\":7,\"failed\":0,\"todo\":61},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locat[...]","@timestamp":"2017-04-26T16:15:02.742897038Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [480a56c5107388dc69db2aff6e997db4] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [480a56c5107388dc69db2aff6e997db4] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":25.92,"view":0.22,"db":9.03,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:02.786600011Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-6h85anwkt785p2d (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [4fda35568454a0e6ee9cba158061361d] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"durat
ion":4.19,"view":0.24,"db":0.51,"params":{},"@timestamp":"2017-04-26T16:15:03.370908997Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78 (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [bf06b3269b0c989254460a78dab3cd41] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-wbmmsy1krcjxecg","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":4.92,"view":0.21,"db":0
.97,"params":{},"@timestamp":"2017-04-26T16:15:03.393788198Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-wbmmsy1krcjxecg (arvados/v1/jobs#show)"}
[api.arvados.sanger] [81afb1b84eafadbc50158f5543856c92] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [81afb1b84eafadbc50158f5543856c92] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":44.72,"view":0.3,"db":33.79,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"44099e6a8666f36b57aa216805c9f03f+1293\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-wbmmsy1krcjxecg\",\"kind\":\"arvados#job\",\"etag\":\"eclfov43onsl809efvn5zq7s9\",\"uuid\":\"z8ta6-8i9sb-wbmmsy1krcjxecg\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-19T01:08:26.601950000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-19T11:03:03.222784000Z\",\"submit_id\":\"instance z8ta6-d1hrv-mhmj4muwtbulu78 rand n6i9ttheutmm\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"44099e6a8666f36b57aa216805c9f03f+1293\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-19T10:22:27.753004000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":129,\"running\":6,\"failed\":1,\"todo\":66},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locato[...]","@timestamp":"2017-04-26T16:15:03.456898482Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [e63a63686654ac830677e671000bbcbd] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e63a63686654ac830677e671000bbcbd] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":12.76,"view":0.13,"db":3.22,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:03.486331868Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-mhmj4muwtbulu78 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [025760ef0366601447c0ba3a00a4c1e2] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.45,"view":0.27,"db":0.77,"params":{},"@timestamp":"2017-04-26T16:15:04.069387399Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [44b43be833391f69bb9d47c50ef9fede] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-46pih4w2evqz1hc","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":4.98,"view":0.23,"db":0.83,"params":{},"@timestamp":"2017-04-26T16:15:04.091572478Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-46pih4w2evqz1hc (arvados/v1/jobs#show)"}
[api.arvados.sanger] [d72ae4382ec7d87dedc6ad7bd13eff4b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d72ae4382ec7d87dedc6ad7bd13eff4b] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":26.5,"view":0.24,"db":15.84,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"71c536ea622448dca8dd43663ba87aa3+874\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-46pih4w2evqz1hc\",\"kind\":\"arvados#job\",\"etag\":\"792wb5x04zrrhum2bhjgqf0ob\",\"uuid\":\"z8ta6-8i9sb-46pih4w2evqz1hc\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-26T02:27:43.238753000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-26T12:10:19.437793000Z\",\"submit_id\":\"instance z8ta6-d1hrv-bk4ncv8a2pubxru rand 12zts4w8z7cz1\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"71c536ea622448dca8dd43663ba87aa3+874\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-26T12:09:50.129258000Z\",\"finished_at\":\"2017-04-26T12:10:19.437793000Z\",\"output\":\"d41d8cd98f00b204e9800998ecf8427e+0\",\"success\":false,\"running\":false,\"state\":\"Failed\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":\"aaaad835431063fd1f8bd6becda9365d+85\",\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":1,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr[...]","@timestamp":"2017-04-26T16:15:04.134677657Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [3b5cc88f3b17c84d78cf6c2a0e75636e] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [3b5cc88f3b17c84d78cf6c2a0e75636e] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":37.78,"view":0.17,"db":27.52,"params":{"pipeline_instance":{"state":"Failed","finished_at":"2017-04-26 17:15:04 +0100","components_summary":{"todo":0,"done":0,"failed":1}}},"@timestamp":"2017-04-26T16:15:04.189081934Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-bk4ncv8a2pubxru (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [a519e7821eec26afa569dbe62e2c123f] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.54,"view":0.26,"db":0.46,"params":{},"@timestamp":"2017-04-26T16:15:04.785286139Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [db921fb2e896f622cee9c5c3d194c910] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-gq8j219eesscc9f","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.83,"view":0.36,"db":1.18,"params":{},"@timestamp":"2017-04-26T16:15:04.809870758Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-gq8j219eesscc9f (arvados/v1/jobs#show)"}
[api.arvados.sanger] [a5e40b51acecc6b37ddd74703f7f819e] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a5e40b51acecc6b37ddd74703f7f819e] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":41.66,"view":0.28,"db":30.75,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\
"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"d6c77
7c034f704b6e7b117f0ceadc50f+1252\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"m
in_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-gq8j219eesscc9f\",\"kind\":\"arvados#job\",\"etag\":\"ept6gz85nnaylf2qjk6tqif59\",\"uuid\":\"z8ta6-8i9sb-gq8j219eesscc9f\",\"
owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:20:32.090743000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:35:25.487595000Z\
",\"submit_id\":\"instance z8ta6-d1hrv-68qsmgowjxg8pbh rand 1ccr5te7ea6pp\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_c
ollection\":\"d6c777c034f704b6e7b117f0ceadc50f+1252\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client
_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:35:24.549887000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2
twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondet
erministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\"[...]","@timestamp":"2017-04-26T16:15:04.868954861Z","@version":"1","message":"[200] PUT /arvados
/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [3819cb4f118c7465b8311fe9a00a9717] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [3819cb4f118c7465b8311fe9a00a9717] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":26.53,"view":0.17,"db":16.49,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:04.918430363Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-68qsmgowjxg8pbh (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [67e3210da201f9a4553eaa22a01e13ce] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.45,"view":0.27,"db":0.5,"params":{},"@timestamp":"2017-04-26T16:15:05.503029722Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [bd6e972fe4b59d1146801fa7fce9a334] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-jc19mqvw1kcptzj","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.54,"view":0.4,"db":1.26,"params":{},"@timestamp":"2017-04-26T16:15:05.530277963Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-jc19mqvw1kcptzj (arvados/v1/jobs#show)"}
[api.arvados.sanger] [f78dc04dea56c2cd9d786a8ab4b19507] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [f78dc04dea56c2cd9d786a8ab4b19507] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":39.59,"view":0.3,"db":28.9,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"426946e764690d8979622d1f88a772e0+1168\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-jc19mqvw1kcptzj\",\"kind\":\"arvados#job\",\"etag\":\"das17bjthlzcls5522ina9vq5\",\"uuid\":\"z8ta6-8i9sb-jc19mqvw1kcptzj\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-19T01:07:24.297874000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-19T11:02:37.657291000Z\",\"submit_id\":\"instance z8ta6-d1hrv-fhobj9ul47xb6id rand 2q20wd4z8ea1u\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"426946e764690d8979622d1f88a772e0+1168\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-19T10:15:34.007789000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":188,\"running\":9,\"failed\":0,\"todo\":4},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locato[...]","@timestamp":"2017-04-26T16:15:05.587279115Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [cae570dc437f23bf5ab8b13696c802e0] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [cae570dc437f23bf5ab8b13696c802e0] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":63.46,"view":0.14,"db":53.77,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:05.667432403Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-fhobj9ul47xb6id (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [b5a9edc20c97507a4b0a6f10700feee8] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-ld4icozdmrsbof7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.31,"view":0.22,"db":0.48,"params":{},"@timestamp":"2017-04-26T16:15:06.242638281Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-ld4icozdmrsbof7 (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [675374f99bab1ecde18073070ac330cf] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-bvtmivfb5wpnwmp","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.5,"view":0.23,"db":1.03,"params":{},"@timestamp":"2017-04-26T16:15:06.265199530Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-bvtmivfb5wpnwmp (arvados/v1/jobs#show)"}
[api.arvados.sanger] [c75f178e6f8bfa692e38c965c291a6dd] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [c75f178e6f8bfa692e38c965c291a6dd] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-ld4icozdmrsbof7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":45.65,"view":0.24,"db":35.36,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"fafb52886654f6f3e5f1556fd34d2296+1084\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-bvtmivfb5wpnwmp\",\"kind\":\"arvados#job\",\"etag\":\"cpcwyzqzcx2idzgo8ya6acrs\",\"uuid\":\"z8ta6-8i9sb-bvtmivfb5wpnwmp\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:16:39.824936000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:16:31.781488000Z\",\"submit_id\":\"instance z8ta6-d1hrv-ld4icozdmrsbof7 rand 2e3folbfvhvu7\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"fafb52886654f6f3e5f1556fd34d2296+1084\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:16:30.940450000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\":[...]","@timestamp":"2017-04-26T16:15:06.327292287Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-ld4icozdmrsbof7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [7f39aa4fc209b5ad27a85f87e9c9e961] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [7f39aa4fc209b5ad27a85f87e9c9e961] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-ld4icozdmrsbof7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":22.2,"view":0.16,"db":12.43,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:06.365871055Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8
ta6-d1hrv-ld4icozdmrsbof7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [ca71808ba3d7453f79d50c8b7dddabd8] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.68,"view":0.28,"db":0.73,"params":{},"@timestamp":"2017-04-26T16:15:06.944759709Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [16fea4e184a36010556bfb0b6b6a28c7] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-aqmd975mfxfmwbe","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.57,"view":0.22,"db":0.98,"params":{},"@timestamp":"2017-04-26T16:15:06.968081660Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-aqmd975mfxfmwbe (arvados/v1/jobs#show)"}
[api.arvados.sanger] [33f17e8aa57982f85d0824281288f93d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [33f17e8aa57982f85d0824281288f93d] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":34.45,"view":0.24,"db":17.6,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"cc4ecdd21560b34b729e5f72e9383d9d+873\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-aqmd975mfxfmwbe\",\"kind\":\"arvados#job\",\"etag\":\"2kbaoun4i8r1um52ftlleyc7n\",\"uuid\":\"z8ta6-8i9sb-aqmd975mfxfmwbe\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:52:55.132345000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:29:53.276454000Z\",\"submit_id\":\"instance z8ta6-d1hrv-tx5vqkgnjgnvspd rand 2oaeyw3zho54u\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"cc4ecdd21560b34b729e5f72e9383d9d+873\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:29:52.427141000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\":\"[...]","@timestamp":"2017-04-26T16:15:07.018877456Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [d055fc47b76793a26fa4529f34413a82] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d055fc47b76793a26fa4529f34413a82] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":61.36,"view":0.14,"db":51.39,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:07.097461687Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-tx5vqkgnjgnvspd (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [29c70cff5a37e42c3c170499824641f9] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":11.37,"view":1.02,"db":1.18,"params":{},"@timestamp":"2017-04-26T16:15:07.699507543Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [2dfa13058a8e7cd8ef9f2ebe35025b42] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-l0jymolpxp5tp4l","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":8.23,"view":0.58,"db":1.21,"params":{},"@timestamp":"2017-04-26T16:15:07.731773984Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-l0jymolpxp5tp4l (arvados/v1/jobs#show)"}
[api.arvados.sanger] [2b82393c5c10ae011c3949d3c02027cb] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [2b82393c5c10ae011c3949d3c02027cb] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":23.78,"view":0.37,"db":11.7,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"e7815ea6ac8c708f596a1385efccdd40+1252\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-l0jymolpxp5tp4l\",\"kind\":\"arvados#job\",\"etag\":\"1pua90hkqrl7xlkbtvyj7z7ss\",\"uuid\":\"z8ta6-8i9sb-l0jymolpxp5tp4l\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:13:46.862268000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T02:38:33.902371000Z\",\"submit_id\":\"instance z8ta6-d1hrv-biuimjf2yv251vj rand 2lzgo0iwv92sx\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"e7815ea6ac8c708f596a1385efccdd40+1252\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T02:38:32.599697000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\"[...]","@timestamp":"2017-04-26T16:15:07.776566526Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [0f2ad2adfe8607296513b6f17aadb4dd] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [0f2ad2adfe8607296513b6f17aadb4dd] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":36.22,"view":0.18,"db":25.67,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:07.831571142Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-biuimjf2yv251vj (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [079a4718d233d372fa1110551a27fcad] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.4,"view":0.26,"db":0.44,"params":{},"@timestamp":"2017-04-26T16:15:08.415516229Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [184d8f0330794ea0b14872ea7427c9f2] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-x6tmtnsxkde9v4m","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.05,"view":0.26,"db":0.97,"params":{},"@timestamp":"2017-04-26T16:15:08.439901586Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-x6tmtnsxkde9v4m (arvados/v1/jobs#show)"}
[api.arvados.sanger] [b37c636d928a60ae63f6bd5036fa0527] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [b37c636d928a60ae63f6bd5036fa0527] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":32.7,"view":0.24,"db":21.87,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"
script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"b3a1da
19a1c1b08a63f8a3c4c2662e34+1083\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"mi
n_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-x6tmtnsxkde9v4m\",\"kind\":\"arvados#job\",\"etag\":\"3x05qbjsguvzex1cclp4bohso\",\"uuid\":\"z8ta6-8i9sb-x6tmtnsxkde9v4m\",\"o
wner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T10:47:07.544297000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-17T19:57:54.852658000Z\"
,\"submit_id\":\"instance z8ta6-d1hrv-emagyvehxgnkwac rand 3b8119rdt7s34\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_co
llection\":\"b3a1da19a1c1b08a63f8a3c4c2662e34+1083\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_
uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-17T19:43:07.391743000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":22,\"running\":9,\"failed\":0,\"todo\":170},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locat[...]","@timestamp":"2017-04-26T16:15:08.490353991Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [63e37676b712345baa67fe1954722213] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [63e37676b712345baa67fe1954722213] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":38.97,"view":0.2,"db":28.39,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:08.546797677Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-emagyvehxgnkwac (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [488650186fec2e38f056910aaf107056] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.1,"view":0.24,"db":0.46,"params":{},"@timestamp":"2017-04-26T16:15:09.123216447Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7 (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [a951ce39a5582775421eb3501df356dd] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-7g2dpernc25htce","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":18.25,"view":0.28,"db":1.06,"params":{},"@timestamp":"2017-04-26T16:15:09.161583047Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-7g2dpernc25htce (arvados/v1/jobs#show)"}
[api.arvados.sanger] [b547521127e3f2929c8b495f3293e3e5] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [b547521127e3f2929c8b495f3293e3e5] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":39.71,"view":0.19,"db":29.1,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4fa333094ea9f2d6d6be94cb30603093+1126\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-7g2dpernc25htce\",\"kind\":\"arvados#job\",\"etag\":\"abr2m57zltua080rzkrhu264n\",\"uuid\":\"z8ta6-8i9sb-7g2dpernc25htce\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T10:45:45.816079000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-17T19:57:54.296685000Z\",\"submit_id\":\"instance z8ta6-d1hrv-vaxsbvcgfb7kce7 rand kl0eosrrgls0\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"4fa333094ea9f2d6d6be94cb30603093+1126\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-17T18:43:47.689258000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":159,\"running\":0,\"failed\":0,\"todo\":42},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locato[...]","@timestamp":"2017-04-26T16:15:09.218525615Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [7b163d26610da08004b5df8806f8861d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [7b163d26610da08004b5df8806f8861d] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":44.46,"view":0.15,"db":34.71,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:09.279577864Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-vaxsbvcgfb7kce7 (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [fa1136d1592b0744fe58c5c30c2776e8] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h9qkfqvv86s43r","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.3,"view":0.29,"db":0.48,"params":{},"@timestamp":"2017-04-26T16:15:09.857158480Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-6h9qkfqvv86s43r (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [01b26f972e0fc2d7765fa665890888db] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-wpdnltbydyim0p1","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.13,"view":0.26,"db":1.06,"params":{},"@timestamp":"2017-04-26T16:15:09.882483301Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-wpdnltbydyim0p1 (arvados/v1/jobs#show)"}
[api.arvados.sanger] [651ea9229dc7ead19af209e79241d6f0] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [651ea9229dc7ead19af209e79241d6f0] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h9qkfqvv86s43r","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":20.48,"view":0.27,"db":10.06,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1edb17f2b0ebe29715c8052dd2322b05+1209\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-wpdnltbydyim0p1\",\"kind\":\"arvados#job\",\"etag\":\"5j3mf07k5gb7bz0lllmi5c0ga\",\"uuid\":\"z8ta6-8i9sb-wpdnltbydyim0p1\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-19T01:09:03.600643000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-19T11:03:17.299823000Z\",\"submit_id\":\"instance z8ta6-d1hrv-6h9qkfqvv86s43r rand 207d9xadpsn1d\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"1edb17f2b0ebe29715c8052dd2322b05+1209\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-19T10:33:17.943017000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":103,\"running\":7,\"failed\":0,\"todo\":91},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locat[...]","@timestamp":"2017-04-26T16:15:09.920196613Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-6h9qkfqvv86s43r (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [3d4b5c9d64cda51def325dc650652920] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [3d4b5c9d64cda51def325dc650652920] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6h9qkfqvv86s43r","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":56.18,"view":0.25,"db":44.21,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:09.995904504Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z
8ta6-d1hrv-6h9qkfqvv86s43r (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [acca3b9db093e61e302f1b8dca35bab6] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":3.92,"view":0.19,"db":0.48,"params":{},"@timestamp":"2017-04-26T16:15:10.572533989Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [880a0aa5fd3660032a18ce3847758929] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-wag4zmifepd1u7d","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":5.29,"view":0.22,"db":0.99,"params":{},"@timestamp":"2017-04-26T16:15:10.595609227Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-wag4zmifepd1u7d (arvados/v1/jobs#show)"}
[api.arvados.sanger] [14c5448cce3c535b88b03dcb268af4fa] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [14c5448cce3c535b88b03dcb268af4fa] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":42.7,"view":0.21,"db":32.49,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"ab232433c28af2c16939a7b565f0e11a+1126\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-wag4zmifepd1u7d\",\"kind\":\"arvados#job\",\"etag\":\"azgzdw04hc4vtwid4pvwpiugx\",\"uuid\":\"z8ta6-8i9sb-wag4zmifepd1u7d\",\"owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-16T11:19:21.535920000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T03:31:42.623979000Z\",\"submit_id\":\"instance z8ta6-d1hrv-udmregj42gtnlpc rand 2por6o7e3d4u2\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_collection\":\"ab232433c28af2c16939a7b565f0e11a+1126\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T03:31:41.060516000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":0,\"running\":0,\"failed\":0,\"todo\":1},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_locator\"[...]","@timestamp":"2017-04-26T16:15:10.654711487Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [bab478a9ea338398165ce7a2d26396d4] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [bab478a9ea338398165ce7a2d26396d4] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":26.9,"view":0.16,"db":17.2,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:15:10.698110048Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-udmregj42gtnlpc (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [ac620d6b655f3e89816786f230472a74]
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds (waited 5.000092554 seconds). The max pool size is currently 5; consider increasing it.):
app/middlewares/arvados_api_token.rb:59:in `call'
Connecting to database specified by database.yml
Websockets disabled
[api.arvados.sanger] [a2e91257975922bab6eb5ecda49d48d5] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d377b1e4fdf6694a918d14a4c5472829] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [708df25cd4c2bf0ae15fae193967eace] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a2e91257975922bab6eb5ecda49d48d5] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-t8lt63kdorffy0z/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":42.85,"view":0.54,"db":29.75,"params":{"ping_secret":"3dq3yylg32nj3hbhjl3eq1neqof86r4g01rtr9oj9d24ujw32h","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.169014193Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-t8lt63kdorffy0z/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [708df25cd4c2bf0ae15fae193967eace] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-1ckqm6rgqizti7z/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":28.13,"view":0.29,"db":17.21,"params":{"ping_secret":"ztxt3599ctdovhw6okc89zjya6qf6d78spt4jvmmu4u62nyhx","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.185119694Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-1ckqm6rgqizti7z/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [d377b1e4fdf6694a918d14a4c5472829] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-8thrz8n7g55nxnw/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":35.35,"view":0.38,"db":23.0,"params":{"ping_secret":"2ucyqohee5tjttajfn9uqvt5tsxzdaau0qb74bgmcvxzir4hml","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.191736984Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-8thrz8n7g55nxnw/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [3a10c8656b27ec696a4e1515176a48b0] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.27,"view":0.23,"db":0.48,"params":{},"@timestamp":"2017-04-26T16:20:01.231660996Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [e30807221617366516051f76beebd497] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-0wix30iholy6eqx","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.15,"view":0.28,"db":1.4,"params":{},"@timestamp":"2017-04-26T16:20:01.257236026Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-0wix30iholy6eqx (arvados/v1/jobs#show)"}
[api.arvados.sanger] [a7907bf6a892fb74745a5d9bbc609643] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [73fe7f0f383bb8c3d67bd9df2974e9c9] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a7907bf6a892fb74745a5d9bbc609643] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"dur
ation":51.37,"view":0.31,"db":40.17,"params_truncated":"{\"pipeline_instance\":{\"components\":{\"gatk-haplotypecaller-cram-gvcf\":{\"script\":\"gatk-haplotypecaller-cram.py\",\"script_version\":\"master\",\"repository\":\"jr17/hgi\",\
"script_parameters\":{\"reference_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"1b447d1aa006cde57df03042736750b9+7894\"},\"inputs_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"d202d
11fc766d5da459ea4bad9293fef+1126\"},\"interval_lists_collection\":{\"required\":true,\"dataclass\":\"Collection\",\"value\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"}},\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"m
in_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"job\":{\"href\":\"/jobs/z8ta6-8i9sb-0wix30iholy6eqx\",\"kind\":\"arvados#job\",\"etag\":\"epbtcy0ctlk1nttz2h8ywi0xo\",\"uuid\":\"z8ta6-8i9sb-0wix30iholy6eqx\",\"
owner_uuid\":\"z8ta6-j7d0g-ovuscadxp8hm0iu\",\"created_at\":\"2017-04-18T14:12:52.801839000Z\",\"modified_by_client_uuid\":null,\"modified_by_user_uuid\":\"z8ta6-tpzed-s6urztfw1f2twvc\",\"modified_at\":\"2017-04-18T21:53:36.568526000Z\
",\"submit_id\":\"instance z8ta6-d1hrv-6ncytsbvhe9ryqs rand 2rz39pzyor6c9\",\"priority\":0,\"script\":\"gatk-haplotypecaller-cram.py\",\"script_parameters\":{\"reference_collection\":\"1b447d1aa006cde57df03042736750b9+7894\",\"inputs_c
ollection\":\"d202d11fc766d5da459ea4bad9293fef+1126\",\"interval_lists_collection\":\"4a72f79fd2ed51ac692bc326fb168f37+10043\"},\"script_version\":\"794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4\",\"cancelled_at\":null,\"cancelled_by_client
_uuid\":null,\"cancelled_by_user_uuid\":null,\"started_at\":\"2017-04-18T21:15:35.026224000Z\",\"finished_at\":null,\"output\":null,\"success\":null,\"running\":true,\"state\":\"Running\",\"is_locked_by_uuid\":\"z8ta6-tpzed-s6urztfw1f2
twvc\",\"log\":null,\"runtime_constraints\":{\"docker_image\":\"mercury/gatk-3.5\",\"min_nodes\":1,\"max_tasks_per_node\":10,\"keep_cache_mb_per_task\":1280},\"tasks_summary\":{\"done\":161,\"running\":10,\"failed\":0,\"todo\":30},\"nondeterministic\":null,\"repository\":\"jr17/hgi\",\"supplied_script_version\":\"master\",\"arvados_sdk_version\":null,\"docker_image_loca[...]","@timestamp":"2017-04-26T16:20:01.325990593Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [f523881c2d943189c63a9fd548e431c6] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [64073eceae16aead1868589afa2546e9] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [73fe7f0f383bb8c3d67bd9df2974e9c9] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-goq0yfgpwmoix02/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":63.08,"view":0.34,"db":53.77,"params":{"ping_secret":"nj23u2db7f15zbny1jj16eonvwdiyv3d2jpvubdw784d2cfg3","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.348808343Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-goq0yfgpwmoix02/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [a864763916cc2ac1f268548ffaed4c5b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [a864763916cc2ac1f268548ffaed4c5b] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-87puy14gjzvr3vh/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":35.63,"view":0.25,"db":27.41,"params":{"ping_secret":"44jsn9c7nuf4i6c620r5dus1y0e93bzsh9gp81r1w1lahxjyuv","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.387800503Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-87puy14gjzvr3vh/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [f523881c2d943189c63a9fd548e431c6] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-onpouqgv8tr4yww/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":52.76,"view":0.44,"db":43.2,"params":{"ping_secret":"4ytreob90yvnvjldwxol7ndcr3knkz740u39n93vri43fb9tww","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.389074343Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-onpouqgv8tr4yww/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [64073eceae16aead1868589afa2546e9] {"method":"PUT","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"update","status":200,"duration":42.09,"view":0.49,"db":29.39,"params":{"pipeline_instance":{"components_summary":{"todo":1,"done":0,"failed":0}}},"@timestamp":"2017-04-26T16:20:01.390633091Z","@version":"1","message":"[200] PUT /arvados/v1/pipeline_instances/z8ta6-d1hrv-6ncytsbvhe9ryqs (arvados/v1/pipeline_instances#update)"}
[api.arvados.sanger] [8d49c6dc833110f6a4be2e91f45a178b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [8d49c6dc833110f6a4be2e91f45a178b] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-eakdzltjtr3h2w0/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":30.13,"view":0.26,"db":21.9,"params":{"ping_secret":"4axvt7vaqhqot96geq08n37r76ijge0eujgezgpvwmvgkif533","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.446437889Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-eakdzltjtr3h2w0/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [07ac74c8f7a7bf5a510dae431330815b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [5d3e2fe6c114625889e9e8c2c85419c8] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [9f6d2549caa3fc9b7a2e78e382d8f2b4] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [188bb790be8e1c3b4788f7fb3d67ce23] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [9f6d2549caa3fc9b7a2e78e382d8f2b4] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-acluu7h95fcy0ae/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":13.15,"view":0.27,"db":3.72,"params":{"ping_secret":"e879l5tpkb1c1k71inet02z31xfe9szen0ovu0yc9csxn2u5q","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.480251868Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-acluu7h95fcy0ae/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [188bb790be8e1c3b4788f7fb3d67ce23] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-1i1v5zotflg26jn/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":18.94,"view":0.27,"db":11.08,"params":{"ping_secret":"2qt7o43l6av3vf514765sscjka2g0rkchx8vg3l8qs3wbhcgpc","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.491004246Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-1i1v5zotflg26jn/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [d609c9381ad007f416f40f1701a4fb2f] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [daf669196c9b4e5bd038c676cccc143f] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [5d3e2fe6c114625889e9e8c2c85419c8] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-h2jiyjx5a5d437p/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":72.0,"view":0.21,"db":64.21,"params":{"ping_secret":"603jdphqf44qr4z7znu81ub1aki2t7m3s1vaus3roidafups2q","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.532542994Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-h2jiyjx5a5d437p/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [07ac74c8f7a7bf5a510dae431330815b] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-0lrjzbq96r4dxc2/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":96.85,"view":5.68,"db":31.44,"params":{"ping_secret":"40l0wzytl8wkw1ftsgsyid9vbxbmqpwtkn5mj8ocic4w0uo6u8","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.545130243Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-0lrjzbq96r4dxc2/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [daf669196c9b4e5bd038c676cccc143f] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-a3o8ihbgt1cw1ng/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":32.88,"view":0.28,"db":24.46,"params":{"ping_secret":"4xa6ydqzmxgherk1gehepv9s4rdrlymuhjmv3c5t6vz3va2v6d","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.561337044Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-a3o8ihbgt1cw1ng/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6ace7e604793a7a9a8f4c3b2a96978ff] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [d609c9381ad007f416f40f1701a4fb2f] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-2ybf3w0s1lxvvyl/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":69.36,"view":0.31,"db":59.68,"params":{"ping_secret":"5wdjor8v4hd3ib2dmm4sxbjtqwf7he2o7yd3tulngsgjo5zly1","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.575848361Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-2ybf3w0s1lxvvyl/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [6ace7e604793a7a9a8f4c3b2a96978ff] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-su5w0k705lpb6v3/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":26.48,"view":0.
11,"db":19.83,"params":{"ping_secret":"tejrdi5nqh3ajrx8e3iumw6dg9ox53ngw0xniypfbyq31nqx1","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.590905317Z","@version":"1","messa
ge":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-su5w0k705lpb6v3/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [efa507c6f138206f0107e53e82940979] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e27e431660f3a73e5f5e1a491b8c5f3d] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [e27e431660f3a73e5f5e1a491b8c5f3d] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ch62ym4wo4gh0xm/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":19.22,"view":0.28,"db":10.51,"params":{"ping_secret":"3s9pr2upha1t3kbwdq6n0gdf9aanl6ymoq8htijxgfwkly9mya","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.642078568Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ch62ym4wo4gh0xm/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [8ded90129a22b08cb0fe3e2444040e08] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [efa507c6f138206f0107e53e82940979] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":44.33,"view":0.18,"db":37.88,"params":{"ping_secret":"19acih9yt5owqkku6afjg8v5166jnrvy717076afgzlpy5xc20","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.644764527Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [89665f3f5d11b4f050f61831c520a946] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [8ded90129a22b08cb0fe3e2444040e08] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-dcjuzp0fzjcceb1/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":29.81,"view":0.26,"db":21.42,"params":{"ping_secret":"3yk403toc4iaq70l5hs5qzb4ug011hkcwpvrqvptofg04rs5q6","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.674412957Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-dcjuzp0fzjcceb1/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [25b8d9513f2ee702ad047a28d0f12e35] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [89665f3f5d11b4f050f61831c520a946] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-kjcae6e2x2ze9wy/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":34.03,"view":0.21,"db":26.22,"params":{"ping_secret":"qndwiq4ny11xix5lz2m0mx7mfuyngmdvf7iiakxp6wq3ernyf","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.708154573Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-kjcae6e2x2ze9wy/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [25b8d9513f2ee702ad047a28d0f12e35] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ztb4ykmctl8bt52/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":40.13,"view":0.26,"db":31.55,"params":{"ping_secret":"h4f9lc7utwx1scwl969cawg7yuh506cxtmd2moa6vv4klygyj","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.719143961Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ztb4ykmctl8bt52/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [7e46d89c0ad05c5d88ba7940e762623b] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [8712bff8b83f968c8dc31440402cf751] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [7e46d89c0ad05c5d88ba7940e762623b] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-sa1q59632vhxov6/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":42.16,"view":0.19,"db":35.77,"params":{"ping_secret":"297ksswrlqaq19pob6e0zndgs4bxs4qn162o01dizzgtolaezg","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.779337567Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-sa1q59632vhxov6/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [8712bff8b83f968c8dc31440402cf751] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-dw0j4kqtbn9b4xo/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":32.29,"view":0.32,"db":23.97,"params":{"ping_secret":"1n05zp5gwfq0gnqsu402wp7bgat18uf4eah03fha2zwmlw7rim","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.787208505Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-dw0j4kqtbn9b4xo/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [7d29dccb297ced54d0c61c42e82458e3] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [13ec00268615750a7f65d5e7f4a70fb2] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [501b353e3ddf4af7fcd74bed3f6b0e31] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [7d29dccb297ced54d0c61c42e82458e3] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-5qd5cdeiza4v6lv/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":27.23,"view":0.19,"db":19.72,"params":{"ping_secret":"692ib9l80dem8lz89rwtroi0pawae8xg7k1g63i9hz7ciiyqup","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.906796904Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-5qd5cdeiza4v6lv/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [13ec00268615750a7f65d5e7f4a70fb2] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-ekzlxvozts92sqm/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":18.52,"view":0.18,"db":10.75,"params":{"ping_secret":"5tapdd6uj8509a5whjt4kshfa1jfrno1byd637nwwdj7nge88v","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.913920642Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-ekzlxvozts92sqm/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [501b353e3ddf4af7fcd74bed3f6b0e31] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-avqsjfg0naek3sw/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":18.64,"view":0.2,"db":11.62,"params":{"ping_secret":"4zjbrw6xmfbltrm8k8ds3rzlbi4ezpze5n16ce2d7b7sldpl29","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-26T16:20:01.919691085Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-avqsjfg0naek3sw/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [2cf25afe29ad2dc2ac700684ea62c945] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [ca3edb1bb2485530d6d1444d3e0bf159] WARNING: Can't verify CSRF token authenticity
[api.arvados.sanger] [ca3edb1bb2485530d6d1444d3e0bf159] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-y79anz2iox0q6j4/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":9.1,"view":0.14,"db":2.63,"params":{"ping_secret":"1m9lossetcrt1v2xotm8vsgiu1bcw23e110w4rd2x6xk0j44g4","total_cpu_cores":"32","total_ram_mb":"257867","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.956467904Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-y79anz2iox0q6j4/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [2cf25afe29ad2dc2ac700684ea62c945] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-5co41k3w77aokje/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":200,"duration":37.03,"view":0.18,"db":30.16,"params":{"ping_secret":"453pd282knmh8nwohnehfhfr09ii9qkn9lp9ux0flxorrzjrbr","total_cpu_cores":"32","total_ram_mb":"257851","total_scratch_mb":"788561"},"@timestamp":"2017-04-26T16:20:01.963226617Z","@version":"1","message":"[200] POST /arvados/v1/nodes/z8ta6-7ekkf-5co41k3w77aokje/ping (arvados/v1/nodes#ping)"}
[api.arvados.sanger] [0c156af9119c69c874fac46323279b4d] {"method":"GET","path":"/arvados/v1/pipeline_instances/z8ta6-d1hrv-2cpfk20huot3k1q","format":"*/*","controller":"arvados/v1/pipeline_instances","action":"show","status":200,"duration":4.09,"view":0.26,"db":0.46,"params":{},"@timestamp":"2017-04-26T16:20:02.013478799Z","@version":"1","message":"[200] GET /arvados/v1/pipeline_instances/z8ta6-d1hrv-2cpfk20huot3k1q (arvados/v1/pipeline_instances#show)"}
[api.arvados.sanger] [f009d2d278c6b3cf0b7214d77d1bc3dc] {"method":"GET","path":"/arvados/v1/jobs/z8ta6-8i9sb-xhzo537firn6415","format":"*/*","controller":"arvados/v1/jobs","action":"show","status":200,"duration":6.36,"view":0.32,"db":1.03,"params":{},"@timestamp":"2017-04-26T16:20:02.040101902Z","@version":"1","message":"[200] GET /arvados/v1/jobs/z8ta6-8i9sb-xhzo537firn6415 (arvados/v1/jobs#show)"}
[api.arvados.sanger] [f8ad0e353c1d8f9219ebaa8101634253] WARNING: Can't verify CSRF token authenticity
postgresql-9.3-main.log:
2017-04-26 17:10:01 BST [8911]: [3-1] user=arvados,db=arvados_production,app=Passenger RubyApp: /var/www/arvados-api/current/public,client=::1 LOG: disconnection: session time: 0:15:00.003 user=arvados database=arvados_production host
=::1 port=19166
2017-04-26 17:10:08 BST [11941]: [149-1] user=,db=,app=,client= LOG: checkpoint starting: time
2017-04-26 17:10:15 BST [11941]: [150-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 65 buffers (0.4%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=6.412 s, sync=0.004 s, total=6.418 s; sync files=113, longest=0.000 s, average=0.000 s
2017-04-26 17:11:07 BST [21848]: [1-1] user=,db=,app=,client= LOG: automatic vacuum of table "arvados_production.public.nodes": index scans: 1
pages: 0 removed, 79 remain
tuples: 79 removed, 43 remain
buffer usage: 2274 hits, 2417 misses, 192 dirtied
avg read rate: 3.160 MB/s, avg write rate: 0.251 MB/s
system usage: CPU 0.00s/0.01u sec elapsed 5.97 sec
2017-04-26 17:11:07 BST [21848]: [2-1] user=,db=,app=,client= LOG: automatic analyze of table "arvados_production.public.nodes" system usage: CPU 0.00s/0.00u sec elapsed 0.23 sec
2017-04-26 17:15:08 BST [11941]: [151-1] user=,db=,app=,client= LOG: checkpoint starting: time
2017-04-26 17:15:14 BST [25064]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=26036
2017-04-26 17:15:14 BST [25064]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production
2017-04-26 17:15:15 BST [25064]: [3-1] user=arvados,db=arvados_production,app=/usr/share/passenger/helper-scripts/rack-preloader.rb,client=::1 LOG: disconnection: session time: 0:00:01.212 user=arvados database=arvados_production host=::1 port=26036
Around this time I also had workbench open, and workbench gave a fiddlesticks error:
Oh... fiddlesticks.
An error occurred when Workbench sent a request to the Arvados API server. Try reloading this page. If the problem is temporary, your request might go through next time. If that doesn't work, the information below can help system administrators track down the problem.
API request URL
https://api.arvados.sanger.ac.uk/arvados/v1/collections
API response
{
":errors":[
"#<ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() SSL SYSCALL error: EOF detected\n: SELECT COUNT(DISTINCT \"collections\".\"id\") FROM \"collections\" WHERE (is_trashed = false)>"
],
":error_token":"1493208347+cc93d884"
}
Report problem or email us if you suspect this is a bug.
Related issues
Updated by Joshua Randall over 7 years ago
This just happened again, and I went pretty soon after to look at the logs.
pg_stat_activity did have an autovacuum in progress that started around the time the troubles started:
16414 | arvados_production | 55772 | 10 | postgres | | | | | 2017-04-28 15:38:50.678031+01 | 2017-04-28 15:38:50.698925+01 | 2017-04-28 15:38:50.698925+01 | 2017-04-28 15:38:50.698926+01 | f | active | autovacuum: VACUUM public.logs (to prevent wraparound)
Approximately 71s after the vacuum started, production.log had an error entry:
[api.arvados.sanger] [4dff29f700fbe9c0a7648716507868f5] WARNING: Can't verify CSRF token authenticity [api.arvados.sanger] [4dff29f700fbe9c0a7648716507868f5] #<ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket() can't get socket descriptor: SELECT "nodes".* FROM "nodes" WHERE "nodes"."uuid" = 'z8ta6-7ekkf-gj3szu33y2eg4ca' LIMIT 1> [api.arvados.sanger] [4dff29f700fbe9c0a7648716507868f5] /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:660:in `block in exec_query' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:659:in `exec_query' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1262:in `select' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/database_statements.rb:18:in `select_all' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `block in select_all' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:75:in `cache_sql' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `select_all' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/querying.rb:38:in `block in find_by_sql' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/explain.rb:41:in `logging_query_plan' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/querying.rb:37:in `find_by_sql' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation.rb:171:in `exec_queries' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation.rb:160:in `block in to_a' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/explain.rb:41:in `logging_query_plan' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation.rb:159:in `to_a' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/finder_methods.rb:381:in `find_first' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/finder_methods.rb:122:in `first' /var/www/arvados-api/current/app/controllers/arvados/v1/nodes_controller.rb:21:in `block in ping' /var/www/arvados-api/current/lib/current_api_client.rb:125:in `block in act_as_system_user' /var/www/arvados-api/current/lib/current_api_client.rb:141:in `act_as_user' /var/www/arvados-api/current/lib/current_api_client.rb:124:in `act_as_system_user' /var/www/arvados-api/current/app/controllers/arvados/v1/nodes_controller.rb:20:in `ping' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/base.rb:167:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rendering.rb:10:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/callbacks.rb:18:in `block in process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:535:in `_run__3204673019480545627__process_action__2456585744717701019__callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `__run_callback' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:81:in `run_callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/callbacks.rb:17:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rescue.rb:29:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications.rb:123:in `block in instrument' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications.rb:123:in `instrument' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/instrumentation.rb:29:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/params_wrapper.rb:207:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/railties/controller_runtime.rb:18:in `process_action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/base.rb:121:in `process' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/rendering.rb:46:in `process' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal.rb:203:in `dispatch' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal.rb:246:in `block in action' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:73:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:73:in `dispatch' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:36:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `each' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:608:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/etag.rb:23:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/conditionalget.rb:35:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/head.rb:14:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/params_parser.rb:21:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/flash.rb:242:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/session/abstract/id.rb:210:in `context' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/session/abstract/id.rb:205:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/cookies.rb:341:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/query_cache.rb:64:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `_run__4446880960546779490__call__3597274187383579083__callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `__run_callback' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:385:in `_run_call_callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:81:in `run_callbacks' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.3.6/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/rack/logger.rb:16:in `block in call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/tagged_logging.rb:22:in `tagged' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/rack/logger.rb:16:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/request_id.rb:22:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/methodoverride.rb:21:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/runtime.rb:17:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/cache/strategy/local_cache.rb:72:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/lock.rb:15:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:140:in `forward' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:147:in `pass' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:159:in `invalidate' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:74:in `call!' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:51:in `call' /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/remote_ip.rb:31:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/engine.rb:484:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/application.rb:231:in `call' /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/railtie/configurable.rb:30:in `method_missing' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:160:in `accept_and_process_next_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads' /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [api.arvados.sanger] [4dff29f700fbe9c0a7648716507868f5] Error 1493390401+965725d4: 422 [api.arvados.sanger] [4dff29f700fbe9c0a7648716507868f5] {"method":"POST","path":"/arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping","format":"*/*","controller":"arvados/v1/nodes","action":"ping","status":422,"duration":5.26,"view":0.83,"db":0.53,"params":{"ping_secret":"19acih9yt5owqkku6afjg8v5166jnrvy717076afgzlpy5xc20","total_cpu_cores":"40","total_ram_mb":"193289","total_scratch_mb":"68302106"},"@timestamp":"2017-04-28T14:40:01.269855845Z","@version":"1","message":"[422] POST /arvados/v1/nodes/z8ta6-7ekkf-gj3szu33y2eg4ca/ping (arvados/v1/nodes#ping)"}
another error followed soon after:
[api.arvados.sanger] [ac397f6b889ad0e1e5fdf3a8d78fc1b9] #<ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: SELECT COUNT(DISTINCT "collections"."id") FROM "collections" WHERE "collections"."owner_uuid" = 'z8ta6-tpzed-s6urztfw1f2twvc' AND (is_trashed = false)>
[api.arvados.sanger] [ac397f6b889ad0e1e5fdf3a8d78fc1b9] /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:660:in `block in exec_query'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:659:in `exec_query'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/postgresql_adapter.rb:1262:in `select'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/database_statements.rb:18:in `select_all'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `block in select_all'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:75:in `cache_sql'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `select_all'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/database_statements.rb:24:in `select_one'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/database_statements.rb:30:in `select_value'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/calculations.rb:250:in `execute_simple_calculation'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/calculations.rb:215:in `perform_calculation'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/calculations.rb:159:in `calculate'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/calculations.rb:162:in `calculate'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/relation/calculations.rb:58:in `count'
/var/www/arvados-api/current/app/controllers/application_controller.rb:476:in `object_list'
/var/www/arvados-api/current/app/controllers/arvados/v1/groups_controller.rb:143:in `block in load_searchable_objects'
/var/www/arvados-api/current/app/controllers/arvados/v1/groups_controller.rb:95:in `each'
/var/www/arvados-api/current/app/controllers/arvados/v1/groups_controller.rb:95:in `load_searchable_objects'
/var/www/arvados-api/current/app/controllers/arvados/v1/groups_controller.rb:35:in `contents'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/base.rb:167:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:601:in `_run__1318258362607091153__process_action__3485520397916161765__callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/callbacks.rb:17:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications.rb:123:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/notifications.rb:123:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/base.rb:121:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/abstract_controller/rendering.rb:46:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal.rb:203:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_controller/metal.rb:246:in `block in action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:73:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:36:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/routing/route_set.rb:608:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/etag.rb:23:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/head.rb:14:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/flash.rb:242:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/session/abstract/id.rb:210:in `context'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/session/abstract/id.rb:205:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/query_cache.rb:64:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-3.2.22.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `_run__1837856766289175462__call__1857036167222924939__callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:405:in `__run_callback'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.3.6/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/rack/logger.rb:16:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/tagged_logging.rb:22:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/rack/logger.rb:16:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/methodoverride.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/runtime.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-3.2.22.5/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.4.7/lib/rack/lock.rb:15:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:140:in `forward'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:249:in `fetch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:189:in `lookup'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:66:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-cache-1.6.1/lib/rack/cache/context.rb:51:in `call'
/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-3.2.22.5/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/engine.rb:484:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/application.rb:231:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-3.2.22.5/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
[api.arvados.sanger] [ac397f6b889ad0e1e5fdf3a8d78fc1b9] Error 1493390431+accc8caf: 422
[api.arvados.sanger] [ac397f6b889ad0e1e5fdf3a8d78fc1b9] {"method":"GET","path":"/arvados/v1/groups/contents","format":"json","controller":"arvados/v1/groups","action":"contents","status":422,"duration":198183.61,"view":0.32,"db":34474.25,"params":{"alt":"json","limit":"9223372036854775807","uuid":"z8ta6-tpzed-s6urztfw1f2twvc","offset":"400000"},"@timestamp":"2017-04-28T14:40:31.922692782Z","@version":"1","message":"[422] GET /arvados/v1/groups/contents (arvados/v1/groups#contents)"}
Updated by Joshua Randall over 7 years ago
postgresql-9-3-main.log during the same time period:
2017-04-28 15:49:36 BST [54606]: [45-1] user=,db=,app=,client= LOG: checkpoints are occurring too frequently (22 seconds apart) 2017-04-28 15:49:36 BST [54606]: [46-1] user=,db=,app=,client= HINT: Consider increasing the configuration parameter "checkpoint_segments". 2017-04-28 15:49:36 BST [54606]: [47-1] user=,db=,app=,client= LOG: checkpoint starting: xlog 2017-04-28 15:49:36 BST [6494]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=33950 2017-04-28 15:49:36 BST [6494]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:36 BST [6502]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=33951 2017-04-28 15:49:36 BST [6502]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:37 BST [6514]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=33958 2017-04-28 15:49:37 BST [6514]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:39 BST [6573]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34010 2017-04-28 15:49:39 BST [6573]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:39 BST [6574]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34011 2017-04-28 15:49:39 BST [6574]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:41 BST [6633]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34073 2017-04-28 15:49:41 BST [6633]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:43 BST [6674]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34103 2017-04-28 15:49:43 BST [6674]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:44 BST [6709]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34146 2017-04-28 15:49:44 BST [6709]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:45 BST [6731]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34156 2017-04-28 15:49:45 BST [6731]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:45 BST [54606]: [48-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 2478 buffers (15.1%); 0 transaction log file(s) added, 0 removed, 3 recycled; write=8.960 s, sync=0.045 s, total=9.017 s; sync files=151, longest=0.005 s, average=0.000 s 2017-04-28 15:49:46 BST [6769]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34188 2017-04-28 15:49:46 BST [6769]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:46 BST [6778]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34191 2017-04-28 15:49:46 BST [6778]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:48 BST [6828]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34236 2017-04-28 15:49:48 BST [6828]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:49 BST [6845]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34247 2017-04-28 15:49:49 BST [6845]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:50 BST [6890]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34288 2017-04-28 15:49:50 BST [6890]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:50 BST [6900]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34297 2017-04-28 15:49:50 BST [6900]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:50 BST [6915]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34304 2017-04-28 15:49:50 BST [6915]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:51 BST [6938]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34323 2017-04-28 15:49:51 BST [6938]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:53 BST [7100]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34472 2017-04-28 15:49:53 BST [7100]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:54 BST [7204]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34525 2017-04-28 15:49:54 BST [7205]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34526 2017-04-28 15:49:54 BST [7206]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34527 2017-04-28 15:49:54 BST [7207]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34528 2017-04-28 15:49:54 BST [7208]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34529 2017-04-28 15:49:54 BST [7209]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34530 2017-04-28 15:49:54 BST [7210]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34531 2017-04-28 15:49:54 BST [7211]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34532 2017-04-28 15:49:54 BST [7212]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34534 2017-04-28 15:49:54 BST [7213]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34533 2017-04-28 15:49:54 BST [7214]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34536 2017-04-28 15:49:55 BST [7213]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7204]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7211]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7206]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7207]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7205]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7210]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7214]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7209]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7208]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7212]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:55 BST [7226]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34542 2017-04-28 15:49:55 BST [7226]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:56 BST [54606]: [49-1] user=,db=,app=,client= LOG: checkpoints are occurring too frequently (20 seconds apart) 2017-04-28 15:49:56 BST [54606]: [50-1] user=,db=,app=,client= HINT: Consider increasing the configuration parameter "checkpoint_segments". 2017-04-28 15:49:56 BST [54606]: [51-1] user=,db=,app=,client= LOG: checkpoint starting: xlog 2017-04-28 15:49:59 BST [54606]: [52-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 3481 buffers (21.2%); 0 transaction log file(s) added, 0 removed, 3 recycled; write=2.031 s, sync=0.075 s, total=2.118 s; sync files=151, longest=0.036 s, average=0.000 s 2017-04-28 15:49:59 BST [7438]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34782 2017-04-28 15:49:59 BST [7438]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:59 BST [7447]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34785 2017-04-28 15:49:59 BST [7447]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:59 BST [7456]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34787 2017-04-28 15:49:59 BST [7456]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:49:59 BST [7470]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=34794 2017-04-28 15:49:59 BST [7470]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:50:15 BST [54606]: [53-1] user=,db=,app=,client= LOG: checkpoints are occurring too frequently (19 seconds apart) 2017-04-28 15:50:15 BST [54606]: [54-1] user=,db=,app=,client= HINT: Consider increasing the configuration parameter "checkpoint_segments". 2017-04-28 15:50:15 BST [54606]: [55-1] user=,db=,app=,client= LOG: checkpoint starting: xlog 2017-04-28 15:50:31 BST [54606]: [56-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 4983 buffers (30.4%); 0 transaction log file(s) added, 0 removed, 3 recycled; write=16.370 s, sync=0.060 s, total=16.431 s; sync files=157, longest=0.023 s, average=0.000 s 2017-04-28 15:50:31 BST [7997]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG: connection received: host=::1 port=35157 2017-04-28 15:50:31 BST [7997]: [2-1] user=arvados,db=arvados_production,app=[unknown],client=::1 LOG: connection authorized: user=arvados database=arvados_production 2017-04-28 15:50:42 BST [54606]: [57-1] user=,db=,app=,client= LOG: checkpoints are occurring too frequently (27 seconds apart) 2017-04-28 15:50:42 BST [54606]: [58-1] user=,db=,app=,client= HINT: Consider increasing the configuration parameter "checkpoint_segments". 2017-04-28 15:50:42 BST [54606]: [59-1] user=,db=,app=,client= LOG: checkpoint starting: xlog 2017-04-28 15:50:50 BST [54606]: [60-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 2724 buffers (16.6%); 0 transaction log file(s) added, 0 removed, 3 recycled; write=8.142 s, sync=0.058 s, total=8.212 s; sync files=173, longest=0.002 s, average=0.000 s
Updated by Joshua Randall over 7 years ago
as suggested by the postgres logs, am now raising the checkpoint settings:
was the default:
#checkpoint_segments = 3 # in logfile segments, min 1, 16MB each #checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0
now:
checkpoint_segments = 256 checkpoint_completion_target = 0.9
Updated by Tom Morris about 7 years ago
- Target version set to Arvados Future Sprints
Updated by Tom Clegg over 6 years ago
- Related to Idea #14004: [Controller] Keep database in tune added
Updated by Peter Amstutz over 3 years ago
- Target version deleted (
Arvados Future Sprints)
Actions