Arvados

h1. Running GATK3 using Arvados

This tutorial demonstrates how to run the GATK Haplotype Caller pipeline using GenomeAnalysisTK-3.2-2 from the Broad Institute. These pipelines currently support GATK version 3.2-2 (md5 3163cbeef8fd50d8cb85096758b801a3) (keep content hash 2e98fdc8e90f4c48a0714b711767c9ce+76). You must obtain your own GATK jar file in order to run this pipeline. You can obtain this software by going to the "Broad Institute's GATK licensing site":https://www.broadinstitute.org/gatk/download/. Further instructions on how to upload your file can be found on the "Arvados documentation page":http://doc.arvados.org/user/tutorials/tutorial-keep.html. If you run into any problems, please contact support@curoverse.com

This tutorial introduces the following Arvados features:

* How to run GATK3 using Arvados

* How to access your pipeline results.

* How to browse and select your input data for GATK3 and submit re-run the pipeline.

# Start at the "Curoverse":https://curoverse.com/ website and click Log In at the top. We currently support all Google / Google Apps accounts for authentication. By simply choosing a Google-based account, your account will be automatically created and redirect to the "Arvados Workbench":https://workbench.qr1hi.arvadosapi.com/.

# In the *Active pipelines* panel, click on the *Run a pipeline...* button. Doing so opens a dialog box titled *Choose a pipeline to run*.

# Select *Demo GATK3 Haplotype Caller Pipeline* and click the *Next: choose inputs* button. Doing so loads a new page to supply the inputs for the pipeline.

# The default inputs from the GATK3 source code repository are already pre-loaded. Click on the *Run* button. The page updates to show you that the pipeline has been submitted to run on the Arvados cluster.

# After the pipeline starts running, you can track its progress by watching log messages from jobs.  This page refreshes automatically.  You will see a complete label under the job the column when the pipeline completes successfully. The current run time of the job in CPU and clock hours is also displayed. You can view individual job details by clicking on the job name.

# Once the job is finished, the output can be viewed to the right of the run time.

# Click on the download button to the right of the file to download your results, or the magnifying glass to quickly view your results.

h2. Uploading data through the web and using it on Arvados

# In your home project, click on the blue *+ Add data* button in the top right.

# Click *Upload files from my computer*

# Click *Choose Files* and choose as many sets of paired end fastq files you would like to run GATK3 on.

# Once you're ready, click *> Start*

# Feel free to rename your Collection so you can remember it later. Click on the pencil icon in the top left corner next to *New collection*

# Once that is uploaded, navigate back to the dashboard and click on *Run a pipeline...* and choose Demo GATK3 Haplotype Caller Pipeline.

# You can change the input by clicking on the *[Choose]* button next to the *Sample FASTQ reads*.

# Click on the dropdown menu, click on your newly-created project, and choose your desired input collection. Click *OK* and *Run* to run GATK3 on your data!

h2. Uploading data through your shell and using it on Arvados

Full documentation can be found "here":http://doc.arvados.org/user/tutorials/tutorial-keep.html

# Install the "Arvados Python SDK":http://doc.arvados.org/sdk/python/sdk-python.html on the system from which you will upload the data (such as your workstation, or a server containing data from your sequencer). Doing so will install the Arvados file upload tool, arv-put.

# To configure the environment with the Arvados instance host name and authentication token, see "here":http://doc.arvados.org/user/reference/api-tokens.html 

# Navigate back to your Workbench dashboard and create a new project by clicking on the Projects dropdown menu and clicking Home. 

# Click on [+ Add a subproject]. Feel free to edit the Project name or description by clicking the pencil to the right of the text.

# To add data, return to your shell, create a folder, and put the two paired-end fastq files you want to upload inside. Use the command arv-put * --project-uuid qr1hi-xxxxx-yyyyyyyyyyyyyyy. The qr1hi tag can be found in the url of your new project. This ensures that all the files you would like to upload are in one collection.

# The output value xxxxxxxxxxxxxxxxxxxx+yyyy is the Arvados collection locator that uniquely describes this file.

# Once that is uploaded, navigate back to the dashboard and click on *Run a pipeline...* and choose Demo GATK3 Haplotype Caller Pipeline.

# You can change the input by clicking on [Choose] next to the *Sample FASTQ reads*.

# Click on the dropdown menu, click on your newly-created project, and choose your desired input collection. Click *OK* and *Run* to run GATK3 on your data!

h3. FAQ

WIP