Bug #2739

Crunch script queues only 101 tasks out of 888

Added by Abram Connelly over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
-
Start date:
05/06/2014
Due date:
% Done:

100%

Estimated time:
Story points:
0.5

Description

I am using the crunch script example 'grep' (https://github.com/curoverse/arvados/blob/master/crunch_scripts/grep) on the collection 'bb0c7327a6f35f5c7ed57dd9daf57704+95209'. There should be 888 files in that collection, but when I try to run the crunch script, it looks like only 101 tasks get queued.

Here is a copy of my pipeline template that I use

{
"name": "stamina frazzling Vernon",
"components": {
"sureties": {
"script_parameters": {
"input": "bb0c7327a6f35f5c7ed57dd9daf57704+95209",
"pattern" : "cagtgagttaacatatttgaagtgcttaaaaATGAGGCT"
},
"script_version": "master",
"repository": "abram",
"script": "test/para/grep"
}
}
}

I use the pipeline instance: qr1hi-d1hrv-ivuscfii9d0zcsq .

On lightning-dev2.shell.arvados, I run the following:

arv pipeline run --no-reuse --run-here --template qr1hi-d1hrv-ivuscfii9d0zcsq

From the output of the above command, I see 101 jobs total that are done, running or todo (none fail). From the output screen from the workbench, I see 100-101 files. Also from the workbench, I see the logs indicate that there are only 100 child tasks.


Related issues

Related to Arvados - Support #2772: Recurring: Support beta usersResolved

Associated revisions

Revision 15945eba (diff)
Added by Brett Smith over 7 years ago

2739: Ensure crunch-job spawns all new tasks.

Without this fix, crunch-job would only spawn tasks up to the API
server default result limit (currently 100). Refs #2739.

Revision 16af0b22
Added by Brett Smith over 7 years ago

Merge branch '2739-fix-100-task-limit-wip'

Closes #2739.

History

#1 Updated by Tom Clegg over 7 years ago

  • Target version set to 2014-05-28 Pipeline Factory

#2 Updated by Tom Clegg over 7 years ago

  • Story points set to 0.5

#3 Updated by Brett Smith over 7 years ago

  • Assigned To set to Brett Smith

#4 Updated by Brett Smith over 7 years ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:16af0b22def5b80a1861788907a2fc45b14e3273.

Also available in: Atom PDF