Project

General

Profile

Cluster configuration » History » Version 2

Tom Clegg, 06/12/2018 08:40 PM

1 1 Tom Clegg
h1. Cluster configuration
2
3
We are (2018) consolidating configuration from per-microservice yaml/json/ini files into a single cluster configuration document that is used by all components.
4
* Long term: system nodes automatically keep their configs synchronized (using something like consul).
5
* Short term: sysadmin uses tools like puppet and terraform to ensure /etc/arvados/config.yml is identical on all system nodes.
6
* Hosts without config files (e.g., hosts outside the cluster) can retrieve the config document from the API server.
7
8
h2. Discovery document
9
10
Previously, we copied selected config values from the API server config into the API discovery document so clients could see them. When clients can get the configuration document itself, this won't be needed. The discovery document should advertise APIs provided by the server, not cluster configuration.
11
12
h2. Example config file
13
14
(Format not yet frozen!)
15
16
<pre><code class="yaml">
17
Clusters:
18
  xyzzy:
19
    BlobSigningKey: ungu355able
20
    BlobSignatureTTL: 172800
21
    Volumes:
22
      xyzzy-keep-0:
23
        Type: s3
24
        Region: us-east
25
        Bucket: xyzzy-keep-0
26
        # [rest of keepstore volume config goes here]
27
    Providers:
28
      AWS:
29
        # [credentials and stuff go here]
30
    InstanceTypes:
31
    - Name: m4.large
32
      VCPUs: 2
33
      RAM: 8000000000
34
      Scratch: 31000000000
35
      Price: 0.1
36
    - Name: m4.large-1t
37
      # same instance type as m4.large but our scripts attach more scratch
38
      ProviderType: m4.large
39
      VCPUs: 2
40
      RAM: 8000000000
41
      Scratch: 999000000000
42
      Price: 0.12
43
    - Name: m4.xlarge
44
      VCPUs: 4
45
      RAM: 16000000000
46
      Scratch: 78000000000
47
      Price: 0.2
48
    - Name: m4.8xlarge
49
      VCPUs: 40
50
      RAM: 160000000000
51
      Scratch: 156000000000
52
      Price: 2
53
    - Name: m4.16xlarge
54
      VCPUs: 64
55
      RAM: 256000000000
56
      Scratch: 310000000000
57
      Price: 3.2
58
    - Name: c4.large
59
      VCPUs: 2
60
      RAM: 3750000000
61
      Price: 0.1
62
    - Name: c4.8xlarge
63
      VCPUs: 36
64
      RAM: 60000000000
65
      Price: 1.591
66
    SystemNodes:
67 2 Tom Clegg
      # Key is a class/type/profile, specified on command line, defaulting to hostname
68
      keep:
69 1 Tom Clegg
        # Don’t run other services automatically -- only specified ones
70
        Default: {Disable: true}
71 2 Tom Clegg
        Keepstore: {Listen: ":25107"}
72 1 Tom Clegg
      apiserver:
73
        Default: {Disable: true}
74 2 Tom Clegg
        RailsAPI: {Listen: ":9000", TLS: true}
75
        Controller: {Listen: ":9100"}
76
        Websocket: {Listen: ":9101"}
77
        Health: {Listen: ":9199"}
78 1 Tom Clegg
      keep:
79
        Default: {Disable: true}
80 2 Tom Clegg
        KeepProxy: {Listen: ":9102"}
81
        KeepWeb: {Listen: ":9103"}
82 1 Tom Clegg
      *:
83
        # This section used for a system node not listed above
84
        Default: {Disable: false} # (this is the default behavior)
85
    PostgreSQL:
86
      Host: localhost
87
      Port: 5432
88
      Username: arvados
89
      Password: s3cr3t
90
      Database: arvados_production
91
      Encoding: utf8
92
    DNS:
93
      # “default” means route according to method/path
94
      xyzzy.arvadosapi.com: default
95
      # “collections” means always route to keep-web
96
      collections.xyzzy.arvadosapi.com: collections
97
      # leading * is a wildcard (longest match wins)
98
      "*--collections.xyzzy.arvadosapi.com": collections
99
      cloud.curoverse.com: workbench
100
      workbench.xyzzy.arvadosapi.com: workbench
101
      "*.xyzzy.arvadosapi.com": default
102 2 Tom Clegg
    HTTPRequestTimeout: 5m
103 1 Tom Clegg
</code></pre>