Bug #11283

duplicate key value violates unique constraint "index_nodes_on_slot_number"

Added by Nico César 9 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
03/17/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

this is part of a 2.6G log file:

e51c5:/var/log/postgresql# ls -lh postgresql-9.3-main.log-20170326 
-rw-r----- 1 postgres adm 2.6G Mar 26 06:49 postgresql-9.3-main.log-20170326
2017-03-16 21:55:37 UTC ERROR:  duplicate key value violates unique constraint "index_nodes_on_slot_number" 
2017-03-16 21:55:37 UTC DETAIL:  Key (slot_number)=(6) already exists.
2017-03-16 21:55:37 UTC STATEMENT:  UPDATE "nodes" SET "last_ping_at" = '2017-03-16 21:55:37.327129', "ip_address" = '10.38.64.23', "first_ping_at" = '2017-03-16 21:55:37.3
27129', "slot_number" = 6, "updated_at" = '2017-03-16 21:55:37.347028', "modified_at" = '2017-03-16 21:55:37.347028', "modified_by_client_uuid" = NULL, "info" = '---
        ec2_instance_id: compute-z10duswzkdoq87w-e51c5
        last_action: Prepared by Node Manager
        ping_secret: 2louyeuuylbqdz1c6swx6fmfm93ocbhcppnb9onn7qgh47k7vr
        ', "properties" = '---
        cloud_node:
          price: 0.149
          size: Standard_D11_v2
        ' WHERE "nodes"."id" = 552

Theory is that this is actually a normal condition (uniqueness gets handled after return), but lazy/chatty logging by Rails.

We should fix this so the pg logs don't fill up the space and/or using CPU to compact the logfile, even if Rails ends up doing the right thing, it is CPU and I/O wasted for no reason. If we can't deal at the Rails level, Could there be a flag to avoid logging them at least?


Subtasks

Task #11859: Review 11283-available-slot-queryResolvedPeter Amstutz

Associated revisions

Revision b89fda19
Added by Tom Clegg 6 months ago

Merge branch '11283-available-slot-query'

closes #11283

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

Revision 2c542c9e (diff)
Added by Tom Clegg 4 months ago

11283: Update api server's arvados gems.

refs #11283

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Morris 9 months ago

  • Project changed from OPS to Arvados
  • Description updated (diff)
  • Target version set to Arvados Future Sprints

#2 Updated by Nico César 9 months ago

  • Description updated (diff)

#3 Updated by Tom Clegg 9 months ago

We can do better than this in source:services/api/app/models/node.rb

      begin
        self.slot_number = try_slot
        begin
          self.save!
          break
        rescue ActiveRecord::RecordNotUnique
          try_slot += 1
        end
        raise "No available node slots" if try_slot == Rails.configuration.max_compute_nodes
      end while true

#4 Updated by Tom Clegg 8 months ago

  • Category set to API
  • Status changed from New to In Progress
  • Assigned To set to Tom Clegg

11283-available-slot-query @ 3bb2699f6d418b508dd8bf35cec06f9f54ac6bba

#5 Updated by Tom Clegg 6 months ago

rebased

11283-available-slot-query @ 050be6a5be43ab503820955dbca2751ca368063c

#6 Updated by Tom Clegg 6 months ago

Fixed for rails4

11283-available-slot-query @ 82de7379de9a488a1e5ace7bb854a4c271c466a8

#7 Updated by Tom Clegg 6 months ago

  • Target version changed from Arvados Future Sprints to 2017-06-21 sprint

#8 Updated by Tom Clegg 6 months ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:b89fda1904ec63d637c43831df66b334f6a78377.

Also available in: Atom PDF