Project

General

Profile

Port a Pipeline » History » Version 21

Nancy Ouyang, 04/17/2015 04:07 PM

1 7 Tom Clegg
{{>toc}}
2
3 1 Nancy Ouyang
h1. Port a Pipeline
4
5
Like any other tool, Arvados requires time to learn. Thus, we don't encourage using Arvados for initial development of analysis pipelines or exploratory research on small subsets of data, when each quick-and-dirty iteration takes minutes on a single machine. But for any computationally-intense work, Arvados offers a lot of benefits.
6
7 8 Nancy Ouyang
Okay, cool, provenance, reproducibility, easily scaling to gigabytes of data and mucho RAM, evaluating existing pipelines like lobSTR quickly.
8 1 Nancy Ouyang
9 8 Nancy Ouyang
But what about if you want to these benefits when running your own pipelines?
10
In other words, how do you **port a pipeline** to Arvados?
11 1 Nancy Ouyang
12
h2. 1. Quick Way
13
14
First, do you just want to parallelize a single bash script?
15
16 9 Nancy Ouyang
Check if you can use @arv-run@. Take this arv-run example, which searches multiple FASTQ files in parallel, and saves the results to Keep through shell redirection:
17
18
    $ arv-run grep -H -n GCTACCAAGTTT \< *.fa \> output.txt
19 1 Nancy Ouyang
20
h3. 1.1 Install arv-run
21
22
See: http://doc.arvados.org/sdk/python/sdk-python.html and http://doc.arvados.org/user/reference/api-tokens.html, or in short below:
23
<pre>
24
$ sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
25
$ sudo pip install --pre arvados-python-client
26
</pre>
27
(Lost? See http://doc.arvados.org/sdk/python/sdk-python.html )
28
29
If you try to use arv run right now, it will complain about some settings your missing. To fix that,
30
31 13 Nancy Ouyang
# Go to http://cloud.curoverse.com
32 1 Nancy Ouyang
# Login with any Google account (you may need to click login a few times if you hit multiple redirects from Google)
33
# Click in the upper right on your account name -> Manage Account
34 15 Nancy Ouyang
... !{height:10em}manage_account.png!:manage_account.png
35 1 Nancy Ouyang
# Optional: While you're here, click "send request for shell access", since that will give you shell access to a VM with all of the Arvados tools pre-installed.
36 14 Nancy Ouyang
1) !{height:10em}send_request.png!:send_request.png 2) !{height:10em}request_sent.png!:request_sent.png 3) !{height:10em}access_granted.png!:access_granted.png 
37 12 Nancy Ouyang
# Copy the lines of text into your terminal, something like
38 1 Nancy Ouyang
<pre>
39
HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'
40
export ARVADOS_API_TOKEN=sekritlongthing
41 12 Nancy Ouyang
export ARVADOS_API_HOST=qr1hi.arvadosapi.com
42 1 Nancy Ouyang
unset ARVADOS_API_HOST_INSECURE
43 17 Nancy Ouyang
</pre> ... !{height:5em}terminal_ssh.png!:terminal_ssh.png
44 14 Nancy Ouyang
# If you want this to persist across reboot, add the above lines to ~/.bashrc or your ~/.bash_profile
45 1 Nancy Ouyang
46
(Lost? See http://doc.arvados.org/user/reference/api-tokens.html )
47
48
h3. 1.2 Submit job to Arvados
49
50
First, check: Does your command work locally?
51
52
    $ grep -H -n GCTACCAAGTTT *.fa
53
54
If so, then submit it to arvados using @arv run@
55
56
    $ arv-run grep -H -n GCTACCAAGTTT \< *.fa \> output.txt
57
58 21 Nancy Ouyang
* This bash command stores the results as @output.txt@
59 1 Nancy Ouyang
* Note that due to the particulars of grep, Arvados will report this pipeline as failed if grep does not find anything, and no output will appear on Arvados
60
61 21 Nancy Ouyang
Your dataset is uploaded to Arvados if it wasn't on there already (which may take a while if it's a large dataset), your @grep@ job is submitted to run on the Arvados cluster, and you get the results in a few minutes (stored inside @output.txt@ in Arvados). If you go to Workbench at http://cloud.curoverse.com, you will see the pipeline running. It may take a few minutes for Arvados to spool up a node, provision it, and run your job. The robots are working hard for you, grab a cup of coffee.
62 1 Nancy Ouyang
63
(Lost? See http://doc.arvados.org/user/topics/arv-run.html )
64
65
h3. 1.3 However
66
67 10 Nancy Ouyang
If your pipeline looks more like
68
69 11 Nancy Ouyang
!{width: 50%}https://arvados.org/attachments/download/428/provenance_graph_full.png!:https://arvados.org/attachments/download/428/provenance_graph_full.png
70 10 Nancy Ouyang
71
@arv-run@ is not powerful enough. Here we gooooo.
72 1 Nancy Ouyang
73 20 Nancy Ouyang
(yes, that is a screenshot of an actual pipeline graph auto-generated by Arvados)
74
75 1 Nancy Ouyang
h2. 2. In Short
76
77
**Estimated reading time: 1 hour.**
78
79
You must write a **pipeline template** that describes your pipeline to Arvados.
80
81
h3. 2.1 VM (Virtual Machine) Access
82
83
Note: You'll need the Arvados set of command-line tools to follow along. The easiest way to get started is to get access to a Virtual Machine (VM) with all the tools pre-installed.
84
85 19 Nancy Ouyang
# Go to http://cloud.curoverse.com
86 1 Nancy Ouyang
# Login with google account (you may need to click login a few times, our redirects are not working well)
87 18 Nancy Ouyang
# Click in the upper right on your account name -> Manage Account
88
# Hit the "Request shell access" button under Manage Account in workbench.
89 1 Nancy Ouyang
90
h3. 2.2 Pipeline Template Example
91
92
Here is what a simple pipeline template looks like, where the output of program A is provided as input to program B. We'll explain what it all means shortly, but first, don't worry -- you'll never be creating a pipeline template from scratch. You'll always copy and modify an existing boilerplate one (yes, a template for the pipeline template! :])
93
94
95
    **pipelinetemplate.json**
96
    {
97
    "name": "Tiny Bash Script",
98
      "components": {
99
        "Create Two Files": {
100
          "script": "run-command",
101
          "script_version": "master",
102
          "repository": "nancy",
103
          "script_parameters": {
104
            "command": [
105
              "$(job.srcdir)/crunch_scripts/createtwofiles.sh"
106
            ]
107
          ,
108
          "runtime_constraints": {
109
            "docker_image": "nancy/cgatools-wormtable"
110
        ,
111
        "Merge Files": {
112
          "script": "run-command",
113
          "script_version": "master",
114
          "repository": "nancy",
115
          "script_parameters": {
116
            "command": [
117
              "$(job.srcdir)/crunch_scripts/mergefiles.sh",
118
              "$(input)"
119
            ],
120
            "input": {
121
              "output_of": "Create Two Files"
122
          ,
123
          "runtime_constraints": {
124
            "docker_image": "nancy/cgatools-wormtable"
125
          
126
h2. 3. simple and sweet port-a-pipeline example
127
128
Okay, let's dig into what's going on.
129
130
h3. 3.1 the setup
131
132
We'll port an artificially simple pipeline which involves just two short bash scripts, described as "A" and "B" below:
133
134
**Script A. Create two files**
135
Input: nothing
136
Output: two files (out1.txt and out2.txt)
137
138
**Script B. Merge two files into a single file**
139
Input: output of step A
140
Output: a single file (output.txt)
141
142
Visually, this looks like [!graph image] (ignore the long strings of gibberish in the rectangles for now).
143
144
Here's what we've explained so far in the pipeline template:
145
146
147
    **pipelinetemplate.json**
148
    {
149
    **"name": "Tiny Bash Script",**
150
      "components": {
151
       **"Create Two Files": {**
152
          "script": "run-command",
153
          "script_version": "master",
154
          "repository": "arvados",
155
          "script_parameters": {
156
            "command": [
157
              "$(job.srcdir)/crunch_scripts/ *createtwofiles.sh* "
158
            ]
159
          ,
160
          "runtime_constraints": {
161
            "docker_image": "nancy/cgatools-wormtable"
162
        ,
163
        **"Merge Files": {**
164
          "script": "run-command",
165
          "script_version": "master",
166
          "repository": "arvados",
167
          "script_parameters": {
168
            "command": [
169
              "$(job.srcdir)/crunch_scripts/ *mergefiles.sh* ",
170
              "$(input)"
171
            ],
172
           **"input": {**
173
              **"output_of": "Create Two Files"**
174
          ,
175
          "runtime_constraints": {
176
            "docker_image": "nancy/cgatools-wormtable"
177
178
179
h3. **3.2 arv-what?**
180
181
Before we go further, let's take a quick step back. Arvados consists of two parts
182
183
**Part 1. Keep** - I have all your files in the cloud!
184
185
You can access your files through your browser, using **Workbench**, or using the Arvados command line (CLI) tools (link: http://doc.arvados.org/sdk/cli/index.html ).
186
187
Visually, this looks like
188
[!image 1: workbench]
189
[!image 2: shell session, arv mount]
190
191
**Part 2. Crunch** - I run all your scripts in the cloud!
192
193
Crunch both dispatches jobs and provides version control for your pipelines.
194
195
You describe your workflow to Crunch using **pipeline templates**. Pipeline templates describe a pipeline ("workflow"), the type of inputs each step in the pipeline requires, and . You provide a high-level description of how data flows through the pipeline—for example, the outputs of programs A and B are provided as input to program C—and let Crunch take care of the details of starting the individual programs at the right time with the inputs you specified.
196
197
[!image 2: complex pipeline]
198
199
Once you save a pipeline template in Arvados, you run it by creating a pipeline instance that lists the specific inputs you’d like to use. The pipeline’s final output(s) will be saved in a project you specify.
200
201
Concretely, a pipeline template describes
202
203
* **data inputs** - specified as Keep content addresses
204
* **job scripts** - stored in a Git version control repository and referenced by a commit hash
205
* **parameters** - which, along with the data inputs, can have default values or can be filled in later when the pipeline is actually run
206
* **the execution environment** - stored in Docker images and referenced by Docker image name
207
208
**What is Docker**? Docker allows Arvados to replicate the execution environment your tools need. You install whatever bioinformatics tools (bwa-mem, vcftools, etc.) you are using inside a Docker image, upload it to Arvados, and Arvados will recreate your environment for computers in the cloud.
209
210
**Protip:** Install stable external tools in Docker. Put your own scripts in a Git repository. This is because each docker image is about 500 GB, so each new docker image takes a while to upload (30 minutes) if you are not using Arvados on a local cluster. In the future, we hope to use small diff files describing just the changes made to Docker image instead of the full Docker image. [Last updated 19 Feb 2015]
211
212
h3. 3.3 In More Detail
213
214
Alright, let's put that all together.
215
216
    **pipelinetemplate.json**
217
    {
218
    "name": "Tiny Bash Script",
219
      "components": {
220
        "Create Two Files": {
221
          "script": "run-command",
222
          "script_version": "master",
223
          "repository": "nancy",
224
          "script_parameters": {
225
            "command": [
226
              "$(job.srcdir)/crunch_scripts/createtwofiles.sh" **#[1]**
227
            ]
228
          ,
229
          "runtime_constraints": {
230
            "docker_image": "nancy/cgatools-wormtable"
231
        ,
232
        "Merge Files": {
233
          "script": "run-command",
234
          "script_version": "master",
235
          "repository": "nancy",
236
          "script_parameters": {
237
            "command": [
238
              "$(job.srcdir)/crunch_scripts/mergefiles.sh", **#[2]**
239
              "$(input)"
240
            ],
241
            "input": {
242
              "output_of": "Create Two Files" **#[3]**
243
          ,
244
          "runtime_constraints": {
245
            "docker_image": "nancy/cgatools-wormtable"    
246
    
247
**Explanation**
248
    
249
[1] **$(job.srcdir)** references the git repository "in the cloud". Even though **run-command** is in nancy/crunch_scripts/ and is "magically found" by Arvados, INSIDE run-command you can't reference other files in the same repo as run-command without this magic variable.
250
251
Any output files as a result of this run-command will be automagically stored to keep as an auto-named collection (which you can think of as a folder for now).
252
253
[2] Okay, so how does the next script know where to find the output of the previous job? run-command will keep track of the collections it's created, so we can feed that in as an argument to our next script. In this "command" section under "run-command", you can think of the commas as spaces. Thus, what this line is saying is "run mergefile.sh on the previous input", or
254
255
  $ mergefiles.sh [directory with output of previous command]
256
257
[3] Here we set the variable "input" to point to the directory with the output of the previous command "Create Two Files".
258
259
(Lost? Try the hands-on example in the next section, or read more detailed documentation on the Arvados website: 
260
261
* http://doc.arvados.org/user/tutorials/running-external-program.html
262
* http://doc.arvados.org/user/topics/run-command.html
263
* http://doc.arvados.org/api/schema/PipelineTemplate.html )
264
265
h3. 3.4 All hands on deck!
266
267
Okay, now that we know the rough shape of what's going on, let's get our hands dirty.
268
269 2 Nancy Ouyang
*From your local machine, login to Arvados virtual machine*
270 1 Nancy Ouyang
271 4 Nancy Ouyang
Single step:
272 1 Nancy Ouyang
273 4 Nancy Ouyang
  nrw@ *@nrw-local@* $ ssh nancy@lightning-dev4.shell.arvados
274
275 1 Nancy Ouyang
(Lost? See "SSH access to machine with Arvados commandline tools installed" http://doc.arvados.org/user/getting_started/ssh-access-unix.html )
276
277
**In VM, create pipeline template**
278
279
A few steps:
280
281
  nancy@ *@lightning-dev4.su92l@* :~$ arv create pipeline_template
282
Created object qr1hi-p5p6p-3p6uweo7omeq9e7
283
$ arv edit qr1hi-p5p6p-3p6uweo7omeq9e7 #Create the pipeline template as described above! [[Todo: concrete thing]]
284
285
(Lost? See "Writing a pipeline template" http://doc.arvados.org/user/tutorials/running-external-program.html )
286
287
*In VM, set up git repository with run_command and our scripts*
288
289 2 Nancy Ouyang
A few steps: 
290 1 Nancy Ouyang
291 2 Nancy Ouyang
  $ mkdir @~@/projects
292
$ cd @~@/projects
293
~/projects $ git clone git@git.qr1hi.arvadosapi.com:nancy.git 
294 1 Nancy Ouyang
295 2 Nancy Ouyang
(Lost? Find your own git URL by going to https://workbench.su92l.arvadosapi.com/manage_account )
296 1 Nancy Ouyang
297 2 Nancy Ouyang
    ⤷Copy run_command & its dependencies into this crunch_scripts
298
  $ git clone https://github.com/curoverse/arvados.git 
299 1 Nancy Ouyang
300 2 Nancy Ouyang
(Lost? Visit https://github.com/curoverse/arvados )
301
302
  @  @$ cd ./nancy
303
  *@~/projects/nancy@* $ mkdir crunch_scripts
304
  *@~/projects/nancy@* $ cd crunch_scripts
305
  *@~/projects/nancy/crunch_scripts@* $ cp @~@/projects/arvados/crunch_scripts/run_command . #trailing dot!
306
  ~/projects/nancy/crunch_scripts$ cp ~/projects/arvados/crunch_scripts/crunchutil . #trailing dot!
307
308
  @  @$ cd ~/projects/nancy/crunch_scripts
309
310
  @  @$ vi createtwofiles.sh
311
    ⤷ $cat createtwofiles.sh
312
    #!/bin/bash
313
    echo "Hello " > out1.txt
314
    echo "Arvados!" > out2.txt
315
316
  @  @$ vi mergefiles.sh
317 1 Nancy Ouyang
    ⤷$cat mergefiles.sh
318 5 Nancy Ouyang
      #!/bin/bash *#[1]*
319
      PREVOUTDIR=$1 *#[2]*
320
      echo $TASK_KEEPMOUNT/by_id/$PREVOUTDIR *#[3]*
321 1 Nancy Ouyang
      cat $TASK_KEEPMOUNT/by_id/$PREVOUTDIR/*.txt > output.txt
322
    
323 5 Nancy Ouyang
⤷ *Explanations*
324 6 Nancy Ouyang
*[1]* We use the @#!@ syntax to let bash know what to execute this file with
325 5 Nancy Ouyang
326
  ⤷To find the location of any particular tool, try using **which**
327
    $ which python
328
    /usr/bin/python
329
    $ which bash
330
    /bin/bash
331
    
332
*[2]* [[TODO: $1]] Here we give a human-readable name, @PREVOUTDIR@, to the first argument given to @mergefiles.sh@, which (referring back to the pipeline template) we defined as the directory containing the output of the previous job (which ran @createtwofiles.sh@).
333
    
334
*[3]* Using the environmental variable @TASK_KEEPMOUNT@ allows us to not make assumptions about where **keep** is mounted. @TASK_KEEPMOUNT@ will be replaced by Arvados automatically with the name of the location to which **keep** is mounted on each worker node. (Lost? Visit http://doc.arvados.org/user/tutorials/tutorial-keep-mount.html )
335 1 Nancy Ouyang
    
336 6 Nancy Ouyang
<pre>$ chmod +x createtwofiles.sh mergefiles.sh # make these files executable</pre>
337 1 Nancy Ouyang
338
**Commit changes and push to remote**
339
340 2 Nancy Ouyang
A few steps: 
341 1 Nancy Ouyang
342 2 Nancy Ouyang
  $ git status #check that everything looks ok
343
$ git add *
344
$ git commit -m “hello world-of-arvados scripts!”
345
$ git push
346
347 1 Nancy Ouyang
**Install Docker**
348
349 2 Nancy Ouyang
A few steps: 
350 1 Nancy Ouyang
351 2 Nancy Ouyang
  $ sudo apt-get install docker.io
352
$ sudo groupadd docker
353
$ sudo gpasswd -a $USER docker #in my case, I replace $USER with “nancy”
354
$ sudo service docker restart
355
$ exec su -l $USER   #if you don’t want to login+out or spawn a new shell, this will restart your shell
356
357 1 Nancy Ouyang
**Make docker less sad about running out of space on the VM**
358
359 2 Nancy Ouyang
A few steps:
360
361
  $ sudo mkdir /data/docker
362
$ sudo vi /etc/default/docker
363
@  @⤷$ cat /etc/default/docker
364
      DOCKER_OPTS="--graph='/data/docker'"
365
      export TMPDIR="/data/docker"
366 1 Nancy Ouyang
     
367
**Make Arvados less sad about running out of space on the VM**
368 2 Nancy Ouyang
369
A few steps: 
370
371 1 Nancy Ouyang
    $ sudo mkdir /data/docker-cache
372 3 Nancy Ouyang
$ sudo chown nancy:nancy /data/docker-cache
373
$ ln -s /data/docker-cache docker
374 1 Nancy Ouyang
375
**Create Docker image with Arvados command-line tools and other tools we want**
376 2 Nancy Ouyang
377
A few steps: 
378
379 3 Nancy Ouyang
    $ docker pull arvados/jobs
380 1 Nancy Ouyang
$ docker run -ti arvados/jobs /bin/bash
381 2 Nancy Ouyang
382 4 Nancy Ouyang
Now we are in the docker image.
383 2 Nancy Ouyang
384 1 Nancy Ouyang
    root@4fa648c759f3:/# apt-get update 
385
386 2 Nancy Ouyang
    @  @⤷In the docker image, install external tools that you don't expect to need to update often. 
387 1 Nancy Ouyang
    For instance, we can install the wormtable python tool in this docker image
388 2 Nancy Ouyang
    @  @# apt-get install libdb-dev
389 3 Nancy Ouyang
    @  @# pip install wormtable
390 1 Nancy Ouyang
391
    @  @  ⤷ Note: If you're installing from binaries, you should either
392 6 Nancy Ouyang
        1) Install in existing places where bash looks for programs (e.g. install in /usr/local/bin/cgatools). 
393
        To see where bash looks, inspect the PATH variable.
394 1 Nancy Ouyang
          #echo $PATH
395
          /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
396 6 Nancy Ouyang
        2) If you put them in a custom directory, remember them to reference them as such in your scripts
397
        (e.g. spell out /home/nrw/local/bin/cgatools).
398
        Arvados will not respect modifyng the $PATH variable by using the ~/.bashrc configuration file in the docker image.
399 2 Nancy Ouyang
400 1 Nancy Ouyang
(Lost? See http://doc.arvados.org/user/topics/arv-docker.html )
401
    
402
  root@4fa648c759f3:/# exit
403 2 Nancy Ouyang
404 1 Nancy Ouyang
*Commit Docker image*
405 2 Nancy Ouyang
<pre>
406 6 Nancy Ouyang
$ docker commit 4fa648c759f3 nancy/cgatools-wormtable #Label the image thoughtfully
407
$ #For instance here I used the name of key tools I installed: cgatools & wormtable
408 2 Nancy Ouyang
</pre>
409 1 Nancy Ouyang
410 2 Nancy Ouyang
*Upload Docker image from your VM to Keep*
411
<pre>
412
$ arv keep docker nancy/cgatools-wormtable #this takes a few minutes
413
$ arv keep docker #lists docker images in the cloud, so you can double-check what was uploaded </pre>
414 1 Nancy Ouyang
415
**Run this pipeline!**
416
Go to Workbench and hit **Run**.
417 2 Nancy Ouyang
<pre>$ firefox http://su92l.arvadosapi.com</pre>
418 1 Nancy Ouyang
[!image: workbench with 'tiny bash script']
419
420 2 Nancy Ouyang
*Note: If no worker nodes are already provisioned, your job may take up to 10 minutes to queue up and start.* Behind-the-scenes, Arvados is requesting compute nodes for you and installing your Docker image and otherwise setting up the environment on those nodes. Then Arvados will be ready to run your job. Be patient -- the wait time may seem frustrating for a trivial pipeline like this, but Arvados really excels at handling long and complicated pipelines with built-in data provenance and pipeline reproducibility.
421 1 Nancy Ouyang
422
h3. 3.5 Celebrate
423
424
Whew! Congratulations on porting your first pipeline to Arvados! Check out http://doc.arvados.org/user/topics/crunch-tools-overview.html to learn more about the different ways to port pipelines to Arvados and how to take full advantage of Arvados's features, like restarting pipelines from where they failed instead of from the beginning. 
425
426
h2. 4. Debugging Tips and Pro-Tips
427
428
h3. **4.1 Pro-tips**
429
430
**Keep mounts are read-only right now. [19 March 2015]**
431
Need to 1) make some temporary directories or 2) change directories away from wherever you started out in but still upload the results to keep?
432
433
For 1, Explicitly use the $HOME directory and make the temporary files there
434
For 2, Use present working directory, $(pwd) at the beginning of your script to write down the directory where run-command will look for files to upload to keep.
435
436
Here's an example:
437
<pre>
438
$ cat mergefiles.sh
439
  TMPDIR=$HOME #directory to make temporary files in
440
  OUTDIR=$(pwd) #directory to put output files in
441
  mkdir $TMPDIR
442
  touch $TMPDIR/sometemporaryfile.txt #this file is deleted when the worker node is stopped
443
  touch $OUTDIR/someoutputfile.txt #this file will be uploaded to keep by run-command
444
</pre>
445
446
* make sure you point to the right repository, your own or arvados.
447
* make sure you pin the script versions of your python sdk, docker image, and script version or you will not get reproducibiltiy.
448
* if you have a file you want to use as a crunch script, make sure its in a crunch_scripts directory. otherwise, arvados will not find it. i.e. ~/path/to/git/repo/crunch_scripts/foo.py
449
450
h3. 4.2 Common log errors and reasons for pipelines to fail
451
452
Todo.
453
454
h3. 4.3 Miscellaneous Notes
455
456
Other ways to avoid the read-only keep mount problem is to use task.vwd which uses symlinks from the output directory which is writable to the colelction in keep. If you can change your working directory to the output directory and do all your work there, you'll avoid the keep read only issue.  (lost? see http://doc.arvados.org/user/topics/run-command.html )
457
    
458
When indexing, i.e. tabix, bwa index, etc. The index file tends to be created in the same directory as your fastq file. In order to avoid this, use ^. There is no way to send the index file to another directory. If you figure out a way, please tell me.
459
460
"bash" "-c" could be your friend, it works sometimes, sometimes it doesnt. I don't have a good handle on why this works sometimes.
461
462
if you're trying to iterate over >1 files using the task.foreach, its important to know that run-command uses a m x n method of making groups. I dont think I can explain it right now, but it may not be exactly what you want and you can trip over it. (lost? see http://doc.arvados.org/user/topics/run-command.html )
463
464
When trying to pair up reads, its hard to use run-command. You have to manipulate basename and hope your file names are foo.1 foo.2. base name will treat the group as foo (because you'll regex the groups as foo) and you can glob for foo.1 and foo.2. but if the file names are foo_1 and foo_2, you cant regex search them for foo becuase you'll get both names into a group and you'll be iterating through both of them twice, because of m x n. 
465
    
466
Your scripts need to point to the right place where the file is. Its currently hard to figure out how to grep the file names, you have to do some magic through the collection api.
467
468
h2. 5. Learn More
469
470
To learn more, head over to the Arvados User Guide documentation online: http://doc.arvados.org/user/