Project

General

Profile

Actions

Feature #8575

closed

Improve performance of crunch-job at reaping batches of children

Added by Joshua Randall about 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
-
Story points:
0.5

Description

When crunch-jobs reaps child tasks (in `reapchildren`) it now does so in batches of all child pids ready for reaping at the time of the call.

However, this proceeds slowly because after each child is reaped, an API call is made to ask the API server if that child has spawned any new tasks. On my system it is limited to processing ~1 task per second because each API call takes about that long to complete.

This can be sped up significantly (especially for short running tasks) by waiting until the end of the batch to do the API server query, and running a single query to check for new tasks that have completed for any of the job tasks that were just reaped.

This change would also require modifying the query so that it selects using `filters` rather than `where` so that a list of "created_by_job_task_uuid" parameters can be given in a single query.

Actions #1

Updated by Joshua Randall about 8 years ago

I've completed this and it is working on my system, but I realise it requires 7263-even-better-busy-behaviour, so I'm inclined to make these changes there since it hasn't been merged yet.

Actions #3

Updated by Tom Clegg about 8 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Brett Smith about 8 years ago

  • Target version set to 2016-03-16 sprint
  • Story points set to 0.5
Actions #5

Updated by Brett Smith about 8 years ago

Review should wait until after #7263/#8099 are merged.

Actions #6

Updated by Brett Smith about 8 years ago

The PR references #7263, which has additional background, so we'll do the work there.

Actions #7

Updated by Brett Smith about 8 years ago

  • Status changed from In Progress to Closed
  • Target version deleted (2016-03-16 sprint)
Actions

Also available in: Atom PDF