Arvados

I ran a 20 node job using GATK tools. Everything was configured to use 4 threads, I set the max number of samples per node to be 4, and the ram that java uses to be 12. I saw about 30 tasks done and 5 failed, so I cancelled the job to investigate why it failed. When searching for traceback, I came across this.

https://cloud.curoverse.com/collections/23fb456eff800b1a712922a26dc6c595+91/qr1hi-8i9sb-7bz8qeymofjmurw.log.txt

2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr Traceback (most recent call last):
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr self.run()
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/events.py", line 70, in run
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr items = self.api.logs().list(limit=1, order="id desc", filters=f).execute()['items']
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr return wrapped(*args, **kwargs)
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 729, in execute
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr raise HttpError(resp, content, uri=self.uri)
2015-07-09_18:18:13 qr1hi-8i9sb-7bz8qeymofjmurw 4595 77 stderr ApiError: <HttpError 504 when requesting https://qr1hi.arvadosapi.com/arvados/v1/logs?alt=json&limit=1&order=id+desc&filters=%5B%5B%22event_type%22%2C+%22in%22%2C+%5B%22create%22%2C+%22update%22%2C+%22delete%22%5D%5D%5D returned "Gateway Time-out">

Before this, I ran it on 8 samples a node with 2 threads a piece with 8 nodes. This time, it told me I didn't give it enough ram, which is why i did what I did above. https://cloud.curoverse.com/jobs/qr1hi-8i9sb-v8ses7qdeb5s76p#Log