Project

General

Profile

Actions

Bug #12107

closed

[API] jobs.create should not do slow git operations

Added by Tom Clegg about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/11/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

It's typical for jobs.create to take ~6 seconds, with the database only accounting for about 75ms.

Our "copy commit to internal.git and tag it" procedure might be responsible for a significant portion of this.

Currently, when creating a new job, source:services/api/app/models/commit.rb does this, whether or not the commit already exists in internal.git:

    must_pipe("echo #{sha1.shellescape}",
              "git --git-dir #{src_gitdir.shellescape} pack-objects -q --revs --stdout",
              "git --git-dir #{dst_gitdir.shellescape} unpack-objects -q")
    must_git(dst_gitdir,
             "tag --force #{tag.shellescape} #{sha1.shellescape}")

Using a copy of an internal.git repo from a production cluster on my workstation, and a commit that is already present in internal.git, I found

git operations time
pack-objects | unpack-objects 4-5s
fetch file:///src_gitdir $sha1 35s including initial auto gc
fetch file:///src_gitdir $sha1 12-14s while auto gc running in background
fetch file:///src_gitdir $sha1 1s after gc done
Proposed improvements:
  • Check whether the commit is already present in internal.git before copying it.
    • [[ `git rev-parse --verify $sha1^{commit}` = $sha1 ]] takes <60ms for both success and fail cases.
  • Use "git fetch file://..." instead of "pack|unpack" to avoid unnecessary work (with the pipe method, pack-objects has to prepare the entire pack whether it's needed or not; fetch is surely smarter).
    • git fetch --no-tags file://$src_gitdir $sha1:refs/tags/$uuid takes ~1s including adding/updating the tag.
  • See if there's a way to do this even more efficiently in cases where the commit isn't yet present. "fetch" takes 1s even when the commit is already present, so perhaps there's another way that finds more shortcuts.

Subtasks 1 (0 open1 closed)

Task #12128: Review 12107-faster-gitResolvedTom Clegg08/11/2017

Actions
Actions

Also available in: Atom PDF