Project

General

Profile

Actions
Copy link

Bug #4499

closed

[SDKs] one_task_per_input_file should call normalize() before getting its list of files

Added by Tim Pierce over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tim Pierce
Category:
SDKs
Target version:
2014-12-10 sprint
Story points:
0.5

Description

sguthrie reports that qr1hi-d1hrv-3n1gezo6gondyrv, despite running with one_task_per_input_file, spawns 300 tasks on an input collection with 60 files in it:

Job qr1hi-8i9sb-zc7ivqn134e8czx takes input collection qr1hi-4zz18-zc07fkmhzhfzriz, which has 64 files.

This job runs addPopulationMemEff.py at commit 5299b46be, which specifies:

arvados.job_setup.one_task_per_input_file(if_sequence=0, and_end_task=True, input_as_path=True)

but the job produces nearly 300 tasks:

        "tasks_summary": {
          "done": 1,
          "running": 8,
          "failed": 0,
          "todo": 292
        },

Subtasks 2 (0 open — 2 closed)

Task #4755: Review 4499-one-task-per-input-file-normalizeResolvedTim Pierce12/09/2014Actions
Task #4619: call normalize before reporting file listResolvedTim Pierce12/09/2014Actions
Actions
Copy link
 #1

Updated by Tim Pierce over 9 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Tim Pierce over 9 years ago

  • Description updated (diff)

It's interesting to me that this collection is kind of degenerate: each of the output files are produced from thousands or millions of chunks from the input blocks. The result is a collection with 60 streams in a 55MB manifest with records up to a million bytes long.

Not yet sure if this is hurting one_task_per_input_file but it seems worth keeping an eye on.

Actions
Copy link
 #3

Updated by Tim Pierce over 9 years ago

  • Subject changed from pipeline creates 300 tasks for job with 64 files to one_task_per_input_file creates 300 tasks for job with 64 files
  • Category set to SDKs
Actions
Copy link
 #4

Updated by Tim Pierce over 9 years ago

  • Subject changed from one_task_per_input_file creates 300 tasks for job with 64 files to [SDKs] one_task_per_input_file creates 300 tasks for job with 64 files
Actions
Copy link
 #5

Updated by Tim Pierce over 9 years ago

Sally reports that re-running this pipeline appears to be producing the expected results, so I recommend that we table this ticket for now and revisit it if we see it happen again.

Actions
Copy link
 #6

Updated by Tom Clegg over 9 years ago

  • Target version changed from Bug Triage to Arvados Future Sprints
Actions
Copy link
 #7

Updated by Tom Clegg over 9 years ago

  • Story points set to 0.5
Actions
Copy link
 #8

Updated by Ward Vandewege over 9 years ago

updated report from Sally:
- running this pipeline on 9tee4 works fine.
- re-running on qr1hi showed the same issue: qr1hi-d1hrv-j1ejo9zu0muvrw7

Actions
Copy link
 #9

Updated by Sarah Guthrie over 9 years ago

Re-appeared on 9tee4 after updating docker image. Update to docker image re-installed the python arvados sdk. Change producing the errant behavior was from e7134f -> d11fe66

sguthrie/numpy latest d11fe661b55e 9tee4-4zz18-yoqdj38cazjmf7l Mon Nov 17 23:51:41 2014
sguthrie/numpy latest e7134f3d7e0d 9tee4-4zz18-hbsoox45ftmw6r7 Thu Nov 13 23:40:07 2014

Problem appeared on pipeline instance: 9tee4-d1hrv-x9d8l2ntz3grvim

Actions
Copy link
 #10

Updated by Tom Clegg over 9 years ago

I'm going out on a limb here and guessing adding cr.normalize() before before this part of one_task_per_input_file will fix it:

        for s in cr.all_streams():
            for f in s.all_files():
Actions
Copy link
 #11

Updated by Sarah Guthrie over 9 years ago

Actions
Copy link
 #12

Updated by Tom Clegg over 9 years ago

  • Target version changed from Arvados Future Sprints to 2014-12-10 sprint
Actions
Copy link
 #13

Updated by Tim Pierce over 9 years ago

  • Assigned To set to Tim Pierce
Actions
Copy link
 #14

Updated by Tom Clegg over 9 years ago

  • Subject changed from [SDKs] one_task_per_input_file creates 300 tasks for job with 64 files to [SDKs] one_task_per_input_file should call normalize() before getting its list of files
Actions
Copy link
 #15

Updated by Tim Pierce over 9 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #16

Updated by Peter Amstutz over 9 years ago

LGTM

Actions
Copy link
 #17

Updated by Tim Pierce over 9 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:181e90a0934d0057202b92010a96139df934aaf1.

Actions
Copy link

Also available in: Atom PDF