Idea #13731: Design new nodemanager config data organization for cluster-wide config - Arvados

Actions

Copy link

Idea #13731

closed

Design new nodemanager config data organization for cluster-wide config

Added by Tom Morris over 6 years ago. Updated over 4 years ago.

Status:

Closed

Priority:

Normal

Assigned To:

Category:

Target version:

Start date:

08/13/2018

Due date:

Story points:

0.5

Description

Add to Cluster configuration

Here's an example of the Puppet Hiera config, this is a yaml file used to populate the current .ini file, so one obvious approach would be to use the heira configuration format in "arvados.yml" instead:

arvados-node-manager:
  Logging:
    level: DEBUG
    apiclient: WARNING
    googleapiclient: WARNING
  Daemon:
    dispatcher: slurm
    max_nodes: 8
    poll_time: 10
    boot_fail_after: 1200
  Arvados:
    token: (redacted)
    timeout: 20
  Cloud:
    provider: ec2
    shutdown_windows: 21, 999999
  Cloud Credentials:
    key: (redacted)
    secret: "(redacted)" 
    region: us-east-1
    timeout: 20
  Cloud Create:
    ping_host: 4xphq.arvadosapi.com
    ex_keyname: ward
    image_id: ami-0a01b48b88d14541e
    subnet_id: subnet-24f5ad62
    security_groups: sg-4fc53e2a
  Cloud List:
    instance-state-name: running
    tag_arvados-class: dynamic-compute
    tag_cluster: 4xphq
  Size m4.medium:
    cores: 1
    scratch: 4000
  Size m4.large:
    cores: 2
    scratch: 32000
  Size m4.xlarge:
    cores: 4
    scratch: 80000

Subtasks 1 (0 open — 1 closed)

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Tom Morris over 6 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Tom Morris over 6 years ago

Status changed from In Progress to New
Story points set to 0.5

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Blocks Feature #13611: Nodemanager reads instance sizes from the system config file to avoid duplication added

Actions

Copy link

Updated by Peter Amstutz over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Target version changed from 2018-07-18 Sprint to 2018-08-01 Sprint

Actions

Copy link

Updated by Lucas Di Pentima over 6 years ago

Assigned To changed from Tom Clegg to Lucas Di Pentima

Actions

Copy link

Updated by Lucas Di Pentima over 6 years ago

Status changed from New to In Progress

Actions

Copy link

#10

Updated by Lucas Di Pentima over 6 years ago

Target version changed from 2018-08-01 Sprint to 2018-08-15 Sprint

Actions

Copy link

#11

Updated by Lucas Di Pentima over 6 years ago

Added nodemanager's config items to the arvados-config.yml example.
You can check the differences here: https://dev.arvados.org/projects/arvados/wiki/Cluster_configuration/diff?utf8=✓&version=13&version_from=12&commit=View+differences (although it doesn't show indentation correctly)

As a summary:

Added [Logging] section as a top-level LoggingLevel key so that we can use default logging levels, maybe with the possibility of adding per-service values too
[Cloud Credentials] is replaced by the Providers key, I expanded it a little so that it's more clean how we can manage regions/services/credentials
Node sizes were already existing as top-level items
Moved all cloud node handling specifics inside a Cloud key on NodeManager's own key