Bug #13351

Benchmark container_request creation and see if there are opportunities for optimization

Added by Peter Amstutz 8 months ago. Updated 8 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Subtasks

Task #13354: ReviewNewLucas Di Pentima


Related issues

Related to Arvados - Bug #13108: arvados-cwl-runner dispatches container requests very slowlyResolved2018-04-06

History

#1 Updated by Peter Amstutz 8 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 8 months ago

  • Status changed from In Progress to New
  • Assigned To set to Peter Amstutz
  • Target version changed from To Be Groomed to 2018-04-25 Sprint

#3 Updated by Peter Amstutz 8 months ago

2018-04-13_20:31:54.96718 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:31:54.96718 AZQ validate_scheduling_parameters (0.0ms)

Container Load (6.7ms)  SELECT  "containers".* FROM "containers" WHERE (command IN ('---

2018-04-13_20:31:55.03942 AZQ set_container (74.0ms)

Container Load (16.8ms)  SELECT  "containers".* FROM "containers" WHERE (auth_uuid='2tlax-gj3su-sq60vfd0xzi13wh')  ORDER BY created_at desc LIMIT 1

2018-04-13_20:31:55.09269 AZQ update_priority (4.7ms)
...
2018-04-13_20:31:55.15251 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:31:55.15252 AZQ validate_scheduling_parameters (0.0ms)
2018-04-13_20:31:55.15258 AZQ set_container (0.0ms)
2018-04-13_20:31:55.18055 AZQ update_priority (4.7ms)
2018-04-13_20:31:55.18059 AZQ finalize_if_needed (0.0ms)
2018-04-13_20:31:55.18115 AZQ finalize_if_needed (88.6ms)

{
  "method": "POST",
  "path": "/arvados/v1/container_requests",
  "format": "json",
  "controller": "Arvados::V1::ContainerRequestsController",
  "action": "create",
  "status": 200,
  "duration": 260.67,
  "view": 8.42,
  "db": 79.73,
  "request_id": "req-bf47idlq8kunh1eb8qoq",
  "client_ipaddr": "172.17.0.1",
  "client_auth": "2tlax-gj3su-sq60vfd0xzi13wh",
  "params": {
    "scheduling_parameters": {},
    "runtime_constraints": {
      "vcpus": 1,
      "ram": 1073741824
    },
    "use_existing": true,
    "secret_mounts": {},
    "mounts": {
      "/tmp": {
        "kind": "tmp",
        "capacity": 1073741824
      },
      "/var/spool/cwl": {
        "kind": "tmp",
        "capacity": 1073741824
      }
    },
    "owner_uuid": null,
    "properties": {},
    "output_ttl": 0,
    "environment": {
      "HOME": "/var/spool/cwl",
      "TMPDIR": "/tmp" 
    },
    "name": "step2",
    "priority": 500,
    "state": "Committed",
    "command": [
      "/bin/sh",
      "-c",
      "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
    ],
    "container_image": "arvados/jobs",
    "output_path": "/var/spool/cwl",
    "cwd": "/var/spool/cwl",
    "alt": "json",
    "container_request": {
      "owner_uuid": null,
      "name": "step2",
      "properties": {},
      "state": "Committed",
      "mounts": {
        "/tmp": {
          "kind": "tmp",
          "capacity": 1073741824
        },
        "/var/spool/cwl": {
          "kind": "tmp",
          "capacity": 1073741824
        }
      },
      "runtime_constraints": {
        "vcpus": 1,
        "ram": 1073741824
      },
      "container_image": "arvados/jobs",
      "environment": {
        "HOME": "/var/spool/cwl",
        "TMPDIR": "/tmp" 
      },
      "cwd": "/var/spool/cwl",
      "command": [
        "/bin/sh",
        "-c",
        "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
      ],
      "output_path": "/var/spool/cwl",
      "priority": 500,
      "use_existing": true,
      "scheduling_parameters": {},
      "output_ttl": 0
    }
  },
  "@timestamp": "2018-04-13T20:31:55.199710949Z",
  "@version": "1",
  "message": "[200] POST /arvados/v1/container_requests (Arvados::V1::ContainerRequestsController#create)" 
}

#4 Updated by Peter Amstutz 8 months ago

  • Status changed from New to In Progress
2018-04-13_20:42:21.61739 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:42:21.61741 AZQ validate_scheduling_parameters (0.0ms)
2018-04-13_20:42:21.65299 AZQ set_container (35.5ms)
2018-04-13_20:42:21.68403 AZQ update_priority (16.2ms)
2018-04-13_20:42:21.68643 AZQ finalize_if_needed (2.3ms)

{
  "method": "POST",
  "path": "/arvados/v1/container_requests",
  "format": "json",
  "controller": "Arvados::V1::ContainerRequestsController",
  "action": "create",
  "status": 200,
  "duration": 78.78,
  "view": 0.28,
  "db": 20.74,
  "request_id": "req-txhhgy81wc2bv6faujc2",
  "client_ipaddr": "172.17.0.1",
  "client_auth": "2tlax-gj3su-sq60vfd0xzi13wh",
  "params": {
    "scheduling_parameters": {},
    "runtime_constraints": {
      "vcpus": 1,
      "ram": 1073741824
    },
    "use_existing": false,
    "secret_mounts": {},
    "mounts": {
      "/tmp": {
        "kind": "tmp",
        "capacity": 1073741824
      },
      "/var/spool/cwl": {
        "kind": "tmp",
        "capacity": 1073741824
      }
    },
    "owner_uuid": null,
    "properties": {},
    "output_ttl": 0,
    "environment": {
      "HOME": "/var/spool/cwl",
      "TMPDIR": "/tmp" 
    },
    "name": "step2",
    "priority": 500,
    "state": "Committed",
    "command": [
      "/bin/sh",
      "-c",
      "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
    ],
    "container_image": "arvados/jobs",
    "output_path": "/var/spool/cwl",
    "cwd": "/var/spool/cwl",
    "alt": "json",
    "container_request": {
      "owner_uuid": null,
      "name": "step2",
      "properties": {},
      "state": "Committed",
      "mounts": {
        "/tmp": {
          "kind": "tmp",
          "capacity": 1073741824
        },
        "/var/spool/cwl": {
          "kind": "tmp",
          "capacity": 1073741824
        }
      },
      "runtime_constraints": {
        "vcpus": 1,
        "ram": 1073741824
      },
      "container_image": "arvados/jobs",
      "environment": {
        "HOME": "/var/spool/cwl",
        "TMPDIR": "/tmp" 
      },
      "cwd": "/var/spool/cwl",
      "command": [
        "/bin/sh",
        "-c",
        "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
      ],
      "output_path": "/var/spool/cwl",
      "priority": 500,
      "use_existing": false,
      "scheduling_parameters": {},
      "output_ttl": 0
    }
  },
  "@timestamp": "2018-04-13T20:42:21.690358193Z",
  "@version": "1",
  "message": "[200] POST /arvados/v1/container_requests (Arvados::V1::ContainerRequestsController#create)" 
}

#5 Updated by Peter Amstutz 8 months ago

2018-04-13_22:01:29.09916 AZQ resolve_container_image (15.1ms)
2018-04-13_22:01:29.13630 AZQ Container.create! (36.8ms)
2018-04-13_22:01:29.13675 AZQ set_container (52.7ms)

#6 Updated by Peter Amstutz 8 months ago

things of note:

  • resolve_container_image and resolve_runtime_constraints are called redundantly, once in resolve() and again in find_reusable().
  • The command field is still being stored in YAML instead of json

Things that seem like candidates for optimization (eg they seem to take longer than the database queries would suggest)

  • resolve_container_image
  • Container.create!

#7 Updated by Peter Amstutz 8 months ago

The lower bound on a create request is something like this (creating an empty "human" record)

{"method":"POST","path":"/arvados/v1/humans","format":"json","controller":"Arvados::V1::HumansContro
ller","action":"create","status":200,"duration":25.56,"view":2.85,"db":9.06,"request_id":"req-dlvx7mko29uh3bermi38","client_ip
addr":"172.17.0.1","client_auth":"2tlax-gj3su-sq60vfd0xzi13wh","params":{"human":"{}","_profile":"true","ensure_unique_name":"
false","help":"false","human_given":"true"},"@timestamp":"2018-04-13T22:58:18.990038797Z","@version":"1","message":"[200] POST
/arvados/v1/humans (Arvados::V1::HumansController#create)"}

#8 Updated by Peter Amstutz 8 months ago

  • Related to Bug #13108: arvados-cwl-runner dispatches container requests very slowly added

#9 Updated by Peter Amstutz 8 months ago

  • Target version changed from 2018-04-25 Sprint to 2018-05-09 Sprint

#10 Updated by Tom Morris 8 months ago

  • Target version changed from 2018-05-09 Sprint to Arvados Future Sprints

Also available in: Atom PDF