Feature #8575
closedImprove performance of crunch-job at reaping batches of children
Description
When crunch-jobs reaps child tasks (in `reapchildren`) it now does so in batches of all child pids ready for reaping at the time of the call.
However, this proceeds slowly because after each child is reaped, an API call is made to ask the API server if that child has spawned any new tasks. On my system it is limited to processing ~1 task per second because each API call takes about that long to complete.
This can be sped up significantly (especially for short running tasks) by waiting until the end of the batch to do the API server query, and running a single query to check for new tasks that have completed for any of the job tasks that were just reaped.
This change would also require modifying the query so that it selects using `filters` rather than `where` so that a list of "created_by_job_task_uuid" parameters can be given in a single query.
Updated by Joshua Randall over 8 years ago
I've completed this and it is working on my system, but I realise it requires 7263-even-better-busy-behaviour, so I'm inclined to make these changes there since it hasn't been merged yet.
Updated by Joshua Randall over 8 years ago
Updated by Brett Smith over 8 years ago
- Target version set to 2016-03-16 sprint
- Story points set to 0.5
Updated by Brett Smith over 8 years ago
Review should wait until after #7263/#8099 are merged.
Updated by Brett Smith over 8 years ago
The PR references #7263, which has additional background, so we'll do the work there.
Updated by Brett Smith over 8 years ago
- Status changed from In Progress to Closed
- Target version deleted (
2016-03-16 sprint)