Project

General

Profile

Actions

Bug #13351

open

Benchmark container_request creation and see if there are opportunities for optimization

Added by Peter Amstutz almost 6 years ago. Updated 27 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Subtasks 1 (1 open0 closed)

Task #13354: ReviewNewActions

Related issues

Related to Arvados - Bug #13108: arvados-cwl-runner dispatches container requests very slowlyResolvedPeter Amstutz04/06/2018Actions
Actions #1

Updated by Peter Amstutz almost 6 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz almost 6 years ago

  • Status changed from In Progress to New
  • Assigned To set to Peter Amstutz
  • Target version changed from To Be Groomed to 2018-04-25 Sprint
Actions #3

Updated by Peter Amstutz almost 6 years ago

2018-04-13_20:31:54.96718 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:31:54.96718 AZQ validate_scheduling_parameters (0.0ms)

Container Load (6.7ms)  SELECT  "containers".* FROM "containers" WHERE (command IN ('---

2018-04-13_20:31:55.03942 AZQ set_container (74.0ms)

Container Load (16.8ms)  SELECT  "containers".* FROM "containers" WHERE (auth_uuid='2tlax-gj3su-sq60vfd0xzi13wh')  ORDER BY created_at desc LIMIT 1

2018-04-13_20:31:55.09269 AZQ update_priority (4.7ms)
...
2018-04-13_20:31:55.15251 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:31:55.15252 AZQ validate_scheduling_parameters (0.0ms)
2018-04-13_20:31:55.15258 AZQ set_container (0.0ms)
2018-04-13_20:31:55.18055 AZQ update_priority (4.7ms)
2018-04-13_20:31:55.18059 AZQ finalize_if_needed (0.0ms)
2018-04-13_20:31:55.18115 AZQ finalize_if_needed (88.6ms)

{
  "method": "POST",
  "path": "/arvados/v1/container_requests",
  "format": "json",
  "controller": "Arvados::V1::ContainerRequestsController",
  "action": "create",
  "status": 200,
  "duration": 260.67,
  "view": 8.42,
  "db": 79.73,
  "request_id": "req-bf47idlq8kunh1eb8qoq",
  "client_ipaddr": "172.17.0.1",
  "client_auth": "2tlax-gj3su-sq60vfd0xzi13wh",
  "params": {
    "scheduling_parameters": {},
    "runtime_constraints": {
      "vcpus": 1,
      "ram": 1073741824
    },
    "use_existing": true,
    "secret_mounts": {},
    "mounts": {
      "/tmp": {
        "kind": "tmp",
        "capacity": 1073741824
      },
      "/var/spool/cwl": {
        "kind": "tmp",
        "capacity": 1073741824
      }
    },
    "owner_uuid": null,
    "properties": {},
    "output_ttl": 0,
    "environment": {
      "HOME": "/var/spool/cwl",
      "TMPDIR": "/tmp" 
    },
    "name": "step2",
    "priority": 500,
    "state": "Committed",
    "command": [
      "/bin/sh",
      "-c",
      "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
    ],
    "container_image": "arvados/jobs",
    "output_path": "/var/spool/cwl",
    "cwd": "/var/spool/cwl",
    "alt": "json",
    "container_request": {
      "owner_uuid": null,
      "name": "step2",
      "properties": {},
      "state": "Committed",
      "mounts": {
        "/tmp": {
          "kind": "tmp",
          "capacity": 1073741824
        },
        "/var/spool/cwl": {
          "kind": "tmp",
          "capacity": 1073741824
        }
      },
      "runtime_constraints": {
        "vcpus": 1,
        "ram": 1073741824
      },
      "container_image": "arvados/jobs",
      "environment": {
        "HOME": "/var/spool/cwl",
        "TMPDIR": "/tmp" 
      },
      "cwd": "/var/spool/cwl",
      "command": [
        "/bin/sh",
        "-c",
        "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
      ],
      "output_path": "/var/spool/cwl",
      "priority": 500,
      "use_existing": true,
      "scheduling_parameters": {},
      "output_ttl": 0
    }
  },
  "@timestamp": "2018-04-13T20:31:55.199710949Z",
  "@version": "1",
  "message": "[200] POST /arvados/v1/container_requests (Arvados::V1::ContainerRequestsController#create)" 
}
Actions #4

Updated by Peter Amstutz almost 6 years ago

  • Status changed from New to In Progress
2018-04-13_20:42:21.61739 AZQ validate_runtime_constraints (0.0ms)
2018-04-13_20:42:21.61741 AZQ validate_scheduling_parameters (0.0ms)
2018-04-13_20:42:21.65299 AZQ set_container (35.5ms)
2018-04-13_20:42:21.68403 AZQ update_priority (16.2ms)
2018-04-13_20:42:21.68643 AZQ finalize_if_needed (2.3ms)

{
  "method": "POST",
  "path": "/arvados/v1/container_requests",
  "format": "json",
  "controller": "Arvados::V1::ContainerRequestsController",
  "action": "create",
  "status": 200,
  "duration": 78.78,
  "view": 0.28,
  "db": 20.74,
  "request_id": "req-txhhgy81wc2bv6faujc2",
  "client_ipaddr": "172.17.0.1",
  "client_auth": "2tlax-gj3su-sq60vfd0xzi13wh",
  "params": {
    "scheduling_parameters": {},
    "runtime_constraints": {
      "vcpus": 1,
      "ram": 1073741824
    },
    "use_existing": false,
    "secret_mounts": {},
    "mounts": {
      "/tmp": {
        "kind": "tmp",
        "capacity": 1073741824
      },
      "/var/spool/cwl": {
        "kind": "tmp",
        "capacity": 1073741824
      }
    },
    "owner_uuid": null,
    "properties": {},
    "output_ttl": 0,
    "environment": {
      "HOME": "/var/spool/cwl",
      "TMPDIR": "/tmp" 
    },
    "name": "step2",
    "priority": 500,
    "state": "Committed",
    "command": [
      "/bin/sh",
      "-c",
      "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
    ],
    "container_image": "arvados/jobs",
    "output_path": "/var/spool/cwl",
    "cwd": "/var/spool/cwl",
    "alt": "json",
    "container_request": {
      "owner_uuid": null,
      "name": "step2",
      "properties": {},
      "state": "Committed",
      "mounts": {
        "/tmp": {
          "kind": "tmp",
          "capacity": 1073741824
        },
        "/var/spool/cwl": {
          "kind": "tmp",
          "capacity": 1073741824
        }
      },
      "runtime_constraints": {
        "vcpus": 1,
        "ram": 1073741824
      },
      "container_image": "arvados/jobs",
      "environment": {
        "HOME": "/var/spool/cwl",
        "TMPDIR": "/tmp" 
      },
      "cwd": "/var/spool/cwl",
      "command": [
        "/bin/sh",
        "-c",
        "'echo' 'starting' && 'sleep' '2' && 'echo' 'the number of the day is' '1'" 
      ],
      "output_path": "/var/spool/cwl",
      "priority": 500,
      "use_existing": false,
      "scheduling_parameters": {},
      "output_ttl": 0
    }
  },
  "@timestamp": "2018-04-13T20:42:21.690358193Z",
  "@version": "1",
  "message": "[200] POST /arvados/v1/container_requests (Arvados::V1::ContainerRequestsController#create)" 
}
Actions #5

Updated by Peter Amstutz almost 6 years ago

2018-04-13_22:01:29.09916 AZQ resolve_container_image (15.1ms)
2018-04-13_22:01:29.13630 AZQ Container.create! (36.8ms)
2018-04-13_22:01:29.13675 AZQ set_container (52.7ms)

Actions #6

Updated by Peter Amstutz almost 6 years ago

things of note:

  • resolve_container_image and resolve_runtime_constraints are called redundantly, once in resolve() and again in find_reusable().
  • The command field is still being stored in YAML instead of json

Things that seem like candidates for optimization (eg they seem to take longer than the database queries would suggest)

  • resolve_container_image
  • Container.create!
Actions #7

Updated by Peter Amstutz almost 6 years ago

The lower bound on a create request is something like this (creating an empty "human" record)

{"method":"POST","path":"/arvados/v1/humans","format":"json","controller":"Arvados::V1::HumansContro
ller","action":"create","status":200,"duration":25.56,"view":2.85,"db":9.06,"request_id":"req-dlvx7mko29uh3bermi38","client_ip
addr":"172.17.0.1","client_auth":"2tlax-gj3su-sq60vfd0xzi13wh","params":{"human":"{}","_profile":"true","ensure_unique_name":"
false","help":"false","human_given":"true"},"@timestamp":"2018-04-13T22:58:18.990038797Z","@version":"1","message":"[200] POST
/arvados/v1/humans (Arvados::V1::HumansController#create)"}

Actions #8

Updated by Peter Amstutz almost 6 years ago

  • Related to Bug #13108: arvados-cwl-runner dispatches container requests very slowly added
Actions #9

Updated by Peter Amstutz almost 6 years ago

  • Target version changed from 2018-04-25 Sprint to 2018-05-09 Sprint
Actions #10

Updated by Tom Morris almost 6 years ago

  • Target version changed from 2018-05-09 Sprint to Arvados Future Sprints
Actions #11

Updated by Peter Amstutz over 2 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #12

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #13

Updated by Peter Amstutz 27 days ago

  • Target version set to Future
Actions

Also available in: Atom PDF