Arvados

(03:36:02 PM) Me: When you did JSON gem evaluations, did you look at Yajl?
(03:36:15 PM) tomclegg: not iirc
(03:36:33 PM) Me: Would you be open to switching, based on those RAM numbers I got?
(03:36:36 PM) tomclegg: 5x sounds better than 52x
(03:36:54 PM) tomclegg: Yes, certainly.

Sorry, but this memory overflow happened many weeks ago, so most state is no longer available. Logs and backtraces had been saved though.

I can't remember exactly, but I suspect that the collection itself was not registered, or not completely.

Also note that the specific error message printed by arv-put says "arv-put: Error creating Collection on project:", but if the collection has actually been created then the message could be changed.

Peter Grandi wrote:

Sorry, but this memory overflow happened many weeks ago, so most state is no longer available. Logs and backtraces had been saved though.

Assuming your API server wasn't super-busy at the time, the timestamps in the logs should be sufficient for you to search for the collection that should exist. Something like:

arv collection list -f '[["created_at", ">", "2016-03-11T07:11:30Z"], ["created_at", "<", "2016-03-11T07:11:35Z"]]'

I can't remember exactly, but I suspect that the collection itself was not registered, or not completely.

This would be really good to double-check using the techniques I suggested. Because if you're right that the collection was not registered correctly in the database, that suggests that the solution we're currently considering is barking up the wrong tree, and wouldn't provide a permanent resolution to this problem.

Also note that the specific error message printed by arv-put says "arv-put: Error creating Collection on project:", but if the collection has actually been created then the message could be changed.

Yes, but that's going to require some support from the API server too. Right now it doesn't distinguish "failed to create collection" from "failed to render created collection" in its response, so the client has no way to distinguish the two cases.

Test case creating a collection with a 35 MiB manifest (represents about 25 TiB).

Master:

Begin post VmHWM: 334872 kB
End post VmHWM:  1770316 kB
23.95 s = .

With decode_www_form_component fix:

Begin post VmHWM: 335260 kB
End post VmHWM:   854340 kB
22.60 s = .

With Oj:

Begin post VmPeak: 2284180 kB
Begin post VmHWM:   335132 kB
End post VmPeak:   2793520 kB
End post VmHWM:     835044 kB

With Yajl:

Begin post VmPeak:  568084 kB
Begin post VmHWM:   371548 kB
End post VmPeak:   1113728 kB
End post VmHWM:     870880 kB

So it looks like this is a Ruby standard library bug:

Rack uses the standard library method URI.decode_www_form_component to process
    parameters.  This method first validates the string with a regular expression,
    and then decodes it using another regular expression.  The bug is in the
    validation; the regular expression that is used generates many backtracking
    points, which results in exponential memory growth when matching large strings.
    The fix is to tweak the validation regex to use "posessive" matching (?>)
    and (.*+) which eliminates backtracking.  The optimized regex requires minimal
    memory and is around 50% faster.

8998-optimize-decode-www-form-component @ d59fa34

Nice find.

Looks like you're not the first -- stdlib 2.2 and 2.3 contain a simpler solution (see http://ruby-doc.org/stdlib-2.2.0/libdoc/uri/rdoc/URI.html#method-c-decode_www_form_component)...

def self.decode_www_form_component(str, enc=Encoding::UTF_8)
  raise ArgumentError, "invalid %-encoding (#{str})" if /%(?!\h\h)/ =~ str
  str.b.gsub(/\+|%\h\h/, TBLDECWWWCOMP_).force_encoding(enc)
end

Suggest wrapping this patch in something like "Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.2')". Also, putting it in middlewares/arvados_api_token.rb seems a little mysterious. Maybe something like initializers/fix_ruby21_decode.rb?

Indentation got weird somehow in the vmpeak func and the new test.

nit: test "test memory usage" can be just test "memory usage" ...

• Yes, Oj mmaps a lot of virtual memory
• Oj uses less real memory (the kind you can run out of) than Yajl
• Ignoring VM mapping, Oj is better than Yajl in both memory economy and speed, therefore we should just stick with Oj

The current theory is the "out of memory" bug here was caused solely by decode_www_form_component, and has nothing to do with the fact that Oj harmlessly maps a lot of memory.

LGTM @ a829e7e