Project

General

Profile

Cluster configuration » History » Revision 4

Revision 3 (Tom Clegg, 06/15/2018 06:33 PM) → Revision 4/33 (Tom Clegg, 06/15/2018 07:12 PM)

h1. Cluster configuration 

 We are (2018) consolidating configuration from per-microservice yaml/json/ini files into a single cluster configuration document that is used by all components. 
 * Long term: system nodes automatically keep their configs synchronized (using something like consul). 
 * Short term: sysadmin uses tools like puppet and terraform to ensure /etc/arvados/config.yml is identical on all system nodes. 
 * Hosts without config files (e.g., hosts outside the cluster) can retrieve the config document from the API server. 

 h2. Discovery document 

 Previously, we copied selected config values from the API server config into the API discovery document so clients could see them. When clients can get the configuration document itself, this won't be needed. The discovery document should advertise APIs provided by the server, not cluster configuration. 

 h2. Example config file 

 (Format not yet frozen!) 

 <pre><code class="yaml"> 
 Clusters: 
   xyzzy: 
     BlobSigningKey: ungu355able 
     BlobSignatureTTL: 172800 
     PostgreSQL: Volumes: 
       Host: localhost xyzzy-keep-0: 
         Type: s3 
         Region: us-east 
         Bucket: xyzzy-keep-0 
         # [rest of keepstore volume config goes here] 
     Providers: 
       Port: 5432 AWS: 
         # [credentials and stuff go here] 
     InstanceTypes: 
     - Name: m4.large 
       Username: arvados VCPUs: 2 
       Password: s3cr3t RAM: 8000000000 
       Database: arvados_production Scratch: 31000000000 
       Encoding: utf8 Price: 0.1 
     HTTPRequestTimeout: 5m - Name: m4.large-1t 
       # same instance type as m4.large but our scripts attach more scratch 
       ProviderType: m4.large 
       VCPUs: 2 
       RAM: 8000000000 
       Scratch: 999000000000 
       Price: 0.12 
     - Name: m4.xlarge 
       VCPUs: 4 
       RAM: 16000000000 
       Scratch: 78000000000 
       Price: 0.2 
     - Name: m4.8xlarge 
       VCPUs: 40 
       RAM: 160000000000 
       Scratch: 156000000000 
       Price: 2 
     - Name: m4.16xlarge 
       VCPUs: 64 
       RAM: 256000000000 
       Scratch: 310000000000 
       Price: 3.2 
     - Name: c4.large 
       VCPUs: 2 
       RAM: 3750000000 
       Price: 0.1 
     - Name: c4.8xlarge 
       VCPUs: 36 
       RAM: 60000000000 
       Price: 1.591 
     NodeProfiles: 
       # Key is a profile name; can be specified on service prog command line, defaults to $(hostname) 
       keep: 
         # Don’t run other services automatically -- only specified ones 
         Default: {Disable: true} 
         Keepstore: {Listen: ":25107"} 
       apiserver: 
         Default: {Disable: true} 
         RailsAPI: {Listen: ":9000", TLS: true} 
         Controller: {Listen: ":9100"} 
         Websocket: {Listen: ":9101"} 
         Health: {Listen: ":9199"} 
       keep: 
         Default: {Disable: true} 
         KeepProxy: {Listen: ":9102"} 
         KeepWeb: {Listen: ":9103"} 
       *: 
         # This section used for a node whose profile name is not listed above 
         Default: {Disable: false} # (this is the default behavior) 
     Volumes: PostgreSQL: 
       xyzzy-keep-0: 
         Type: s3 
         Region: us-east 
         Bucket: xyzzy-keep-0 
         # [rest of keepstore volume config goes here] 
     Providers: Host: localhost 
       AWS: 
         # [credentials and stuff go here] Port: 5432 
       Username: arvados 
       Password: s3cr3t 
       Database: arvados_production 
       Encoding: utf8 
     WebRoutes: 
       # “default” means route according to method/path 
       xyzzy.arvadosapi.com: default 
       # “collections” means always route to keep-web 
       collections.xyzzy.arvadosapi.com: collections 
       # leading * is a wildcard (longest match wins) 
       "*--collections.xyzzy.arvadosapi.com": collections 
       cloud.curoverse.com: workbench 
       workbench.xyzzy.arvadosapi.com: workbench 
       "*.xyzzy.arvadosapi.com": default 
     InstanceTypes: 
     - Name: m4.large 
       VCPUs: 2 
       RAM: 8000000000 
       Scratch: 31000000000 
       Price: 0.1 
     - Name: m4.large-1t 
       # same instance type as m4.large but our scripts attach more scratch 
       ProviderType: m4.large 
       VCPUs: 2 
       RAM: 8000000000 
       Scratch: 999000000000 
       Price: 0.12 
     - Name: m4.xlarge 
       VCPUs: 4 
       RAM: 16000000000 
       Scratch: 78000000000 
       Price: 0.2 
     - Name: m4.8xlarge 
       VCPUs: 40 
       RAM: 160000000000 
       Scratch: 156000000000 
       Price: 2 
     - Name: m4.16xlarge 
       VCPUs: 64 
       RAM: 256000000000 
       Scratch: 310000000000 
       Price: 3.2 
     - Name: c4.large 
       VCPUs: 2 
       RAM: 3750000000 
       Price: 0.1 
     - Name: c4.8xlarge 
       VCPUs: 36 
       RAM: 60000000000 
       Price: 1.591 HTTPRequestTimeout: 5m 
 </code></pre>