Project

General

Profile

Actions

Feature #8575

closed

Improve performance of crunch-job at reaping batches of children

Added by Joshua Randall about 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
-
Story points:
0.5

Description

When crunch-jobs reaps child tasks (in `reapchildren`) it now does so in batches of all child pids ready for reaping at the time of the call.

However, this proceeds slowly because after each child is reaped, an API call is made to ask the API server if that child has spawned any new tasks. On my system it is limited to processing ~1 task per second because each API call takes about that long to complete.

This can be sped up significantly (especially for short running tasks) by waiting until the end of the batch to do the API server query, and running a single query to check for new tasks that have completed for any of the job tasks that were just reaped.

This change would also require modifying the query so that it selects using `filters` rather than `where` so that a list of "created_by_job_task_uuid" parameters can be given in a single query.

Actions

Also available in: Atom PDF