Project

General

Profile

Writing a Script Calling a Third Party Tool » History » Version 6

Sarah Guthrie, 04/06/2016 08:11 PM

1 1 Sarah Guthrie
{{>toc}}
2
3
h1. Writing a Script Calling a Third Party Tool
4
5
Case study: FastQC
6
7
Good tips include:
8
* Keep the Dockerfile in the git repository
9
10
h3. Writing a Dockerfile
11
12
Docker has some wonderful documentation for building Dockerfiles:
13
* A reference for Dockerfiles: https://docs.docker.com/engine/reference/builder/
14
* Dockerfile best practices: https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
15
16
From Docker:
17
18
"""
19
Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.
20
21
This page describes the commands you can use in a Dockerfile. When you are done reading this page, refer to the Dockerfile Best Practices (https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/) for a tip-oriented guide.
22
"""
23
24
<pre>
25
FROM arvados/jobs
26
27
USER root
28
29
RUN apt-get -q update && apt-get -qy install \
30
  fontconfig \
31
  openjdk-6-jre-headless \
32
  perl \
33
  unzip \
34
  wget
35
36
USER crunch
37
38
RUN mkdir /home/crunch/fastqc
39
RUN cd /home/crunch/fastqc && \
40
    wget --quiet http://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.4.zip && \
41
    unzip /home/crunch/fastqc/fastqc_v0.11.4.zip
42
43
</pre>
44
45
h3. How to build a docker image from a Dockerfile
46
47
<pre>
48
docker build -t username/imagename path/to/Dockerfile/
49
</pre>
50
51
h3. How to upload a docker image to Arvados
52
53
<pre>
54
arv keep put username/imagename
55
</pre>
56
57
h3. How to call an external tool from a crunch script
58
59 3 Sarah Guthrie
We strongly recommend using the @subprocess@ module for calling external tools. If the output is small and written to standard out, using @subprocess.check_output@ will ensure the tool completed successfully and return the standard output.
60 1 Sarah Guthrie
61
<pre>
62
import subprocess
63 2 Sarah Guthrie
foo = subprocess.check_output(['echo','foo'])
64 1 Sarah Guthrie
</pre>
65
66 3 Sarah Guthrie
If the output is big, @subprocess.check_call@ can redirect it to a file while ensuring the tool completed successfully.
67 1 Sarah Guthrie
68
<pre>
69
import subprocess
70
with open('foo', 'w') as outfile:
71
    subprocess.check_call(['head', '-c', '1234567', '/dev/urandom'], stdout=outfile)
72
</pre>
73 2 Sarah Guthrie
74
FastQC writes to the current output directory or the output directory specified by the @-o@ flag, so we can use @subprocess.check_call@
75
76
<pre>
77
import subprocess
78
import arvados
79
80
#Grab the file path pointing to the file to run fastqc on 
81
fastq_file = arvados.getjobparam('input_fastq_file')
82
83
cmd = ['perl', '/home/crunch/fastqc/FastQC/fastqc', fastq_file]
84
subprocess.check_call(cmd)
85
</pre>
86 1 Sarah Guthrie
87
h3. Where to put temporary files
88 6 Sarah Guthrie
89 5 Sarah Guthrie
<pre>
90
import arvados
91
92
task = arvados.current_task()
93
tmpdir = task.tmpdir
94
</pre>
95
96
Inside the code:
97 1 Sarah Guthrie
98
<pre>
99 4 Sarah Guthrie
import subprocess
100 1 Sarah Guthrie
import arvados
101 4 Sarah Guthrie
102 1 Sarah Guthrie
task = arvados.current_task()
103
tmpdir = task.tmpdir
104
105 4 Sarah Guthrie
#Grab the file path pointing to the file to run fastqc on 
106
fastq_file = arvados.getjobparam('input_fastq_file')
107
108
cmd = ['perl', '/home/crunch/fastqc/FastQC/fastqc', fastq_file, '-o', tmpdir]
109
subprocess.check_call(cmd)
110
111 1 Sarah Guthrie
</pre>
112
113
h3. How to write data directly to Keep (Using TaskOutputDir)
114
115
<pre>
116
import arvados
117
import arvados.crunch
118
import os
119
outdir = arvados.crunch.TaskOutputDir()
120
121
with open(os.path.join(outdir.path, 'foo'), 'w') as outfile:
122
    subprocess.check_call(['head', '-c', '1234567', '/dev/urandom'], stdout=outfile)
123
124
arvados.task_set_output(outdir.manifest_text())
125
</pre>
126
127
h3. When TaskOutputDir is not the correct choice
128
129
* If the tool writes symbolic links or named pipes, which are not supported by fuse
130
* If the I/O access patterns are not performant with fuse
131
** This occurs in Tophat, which opens 20 file handles on multiple files that it writes out
132
 
133
<pre>
134
import arvados
135
import os
136
task = arvados.current_task()
137
tmpdir = task.tmpdir
138
139
os.mkdir(os.path.join(tmpdir, 'out'))
140
141
with open(os.path.join(tmpdir, 'out', 'foo.txt'), 'w') as out:
142
    subprocess.check_call(['head', '-c', '1234567', '/dev/urandom'], stdout=outfile)
143
144
collection_writer = arvados.collection.CollectionWriter()
145
collection_writer.write_file('random_file.txt')
146
collection_writer.write_directory_tree(os.path.join(tmpdir, 'out'))
147
arvados.task_set_output(collection_writer.finish())
148
149
</pre>
150
151
h3. Putting it all together
152
153
<pre>
154
import subprocess
155
156
cmd = ['perl', '/home/crunch/fastqc/FastQC/fastqc']
157
fq_files = sorted(glob.glob('*.fq*'))
158
fastq_files = sorted(glob.glob('*.fastq*'))
159
cmd.extend(fq_files+fastq_files)
160
cmd.extend(['-o', outdirpath, '-t', str(num_threads)])
161
fastqc_pipe = subprocess.Popen(cmd)
162
fastqc_pipe.wait()
163
164
coll_writer = arvados.CollectionWriter()
165
coll_writer.write_directory_tree(outdirpath)
166
pdh = coll_writer.finish()
167
168
body = {'output':pdh, 'success':fastqc_pipe.returncode==0, 'progress':1.0}
169
arvados.api('v1').job_tasks().update(uuid=this_task['uuid'], body=body).execute()
170
</pre>