Project

General

Profile

Bug #12107

Updated by Tom Clegg over 6 years ago

It's typical for jobs.create to take ~6 seconds, with the database only accounting for about 75ms. 

 Our "copy commit to internal.git and tag it" procedure might be responsible for a significant portion of this. 

 Currently, when creating a new job, source:services/api/app/models/commit.rb does this, whether or not the commit already exists in internal.git: 

 <pre><code class="ruby"> 
     must_pipe("echo #{sha1.shellescape}", 
               "git --git-dir #{src_gitdir.shellescape} pack-objects -q --revs --stdout", 
               "git --git-dir #{dst_gitdir.shellescape} unpack-objects -q") 
     must_git(dst_gitdir, 
              "tag --force #{tag.shellescape} #{sha1.shellescape}") 
 </code></pre> 

 Using a copy of an internal.git repo from a production cluster on my workstation, and a commit that is already present in internal.git, I found 

 After a git gc: 
 |git operations                         |time    || 
 |pack-objects &#x7c; unpack-objects     |4-5s    || 
 |fetch file:///src_gitdir $sha1         |35s     |including initial auto gc            | 
 |fetch file:///src_gitdir $sha1         |12-14s|while auto gc running in background| 
 |fetch file:///src_gitdir $sha1         |3s      |after gc done                        | 

 Proposed improvements: 
 * Check whether the commit is already present in internal.git before copying it ("git rev-list -n1 ${sha1}" takes &lt;60ms for both success and fail cases). 
 * Figure out how to accomplish this more efficiently in cases where the commit isn't yet present.

Back