Cluster configuration » History » Revision 15
Revision 14 (Tom Clegg, 09/05/2018 05:31 PM) → Revision 15/33 (Tom Clegg, 10/03/2018 01:23 PM)
h1. Cluster configuration
We are (2018) consolidating configuration from per-microservice yaml/json/ini files into a single cluster configuration document that is used by all components.
* Long term: system nodes automatically keep their configs synchronized (using something like consul).
* Short term: sysadmin uses tools like puppet and terraform to ensure /etc/arvados/config.yml is identical on all system nodes.
* Hosts without config files (e.g., hosts outside the cluster) can retrieve the config document from the API server.
h2. Discovery document
Previously, we copied selected config values from the API server config into the API discovery document so clients could see them. When clients can get the configuration document itself, this won't be needed. The discovery document should advertise APIs provided by the server, not cluster configuration.
h2. Secrets
Secrets like BlobSigningKey can be given literally in the config file (convenient for dev/test, consul-template, etc) or indirectly using a secret backend. Anticipated backends:
* <code class="yaml">BlobSigningKey: foobar</code> ⇒ the secret is literally <code>foobar</code>
* <code class="yaml">BlobSigningKey: "vault:foobar"</code> ⇒ the secret can be obtained from vault using the vault key "foobar"
* <code class="yaml">BlobSigningKey: "file:/foobar"</code> ⇒ the secret can be read from the local file @/foobar@
* <code class="yaml">BlobSigningKey: "env:FOOBAR"</code> ⇒ the secret can be read from the environment variable @FOOBAR@
h2. Example config file
(Format not yet frozen!)
<pre><code class="yaml">
Clusters:
xyzzy:
BlobSigningKey: ungu355able
BlobSignatureTTL: 172800
SessionKey: 186005aa54cab1ca95a3738e6e954e0a35a96d3d13a8ea541f4156e8d067b4f3
PostgreSQL:
ConnectionPool: 32 # max concurrent connections per arvados server daemon
Connection:
# All parameters here are passed to the PG client library in a connection string;
# see https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS
Host: localhost
Port: 5432
User: arvados
Password: s3cr3t
DBName: arvados_production
client_encoding: utf8
fallback_application_name: arvados
HTTPRequestTimeout: 5m
Defaults:
CollectionReplication: 2
TrashLifetime: 2w
UserActivation:
ActivateNewUsers: true
AutoAdminUser: root@example.com
UserProfileNotificationAddress: notify@example.com
NewUserNotificationRecipients: {}
NewInactiveUserNotificationRecipients: {}
RequestLimits: Limits:
MaxRequestLogParamsSize: 2KB
MaxRequestSize: 128MiB
MaxIndexDatabaseRead: 128MiB
MaxItemsPerResponse: 1000
MultiClusterRequestConcurrency: 4
LogLevel: info
CloudVMs:
BootTimeout: 20m
Driver: Amazon
DriverParameters:
Region: us-east-1
APITimeout: 20s
EC2Key: abcdef
EC2Secret: abcdefghijklmnopqrstuvwxyz
StorageKey: abcdef
StorageSecret: abcdefghijklmnopqrstuvwxyz
ImageID: ami-0a01b48b88d14541e
SubnetID: subnet-24f5ae62
SecurityGroups: sg-3ec53e2a
AuditLogs:
MaxAge: 2w
DeleteBatchSize: 100000
UnloggedAttributes: {} # example: {"manifest_text": true}
ContainerLogStream:
BatchSize: 4KiB
BatchTime: 1s
ThrottlePeriod: 1m
ThrottleThresholdSize: 64KiB
ThrottleThresholdLines: 1024
TruncateSize: 64MiB
PartialLineThrottlePeriod: 5s
Timers:
TrashSweepInterval: 60s
ContainerDispatchPollInterval: 10s
APIRequestTimeout: 20s
Scaling:
MaxComputeNodes: 64
EnablePreemptibleInstances: false
DisableAPIMethods: {} # example: {"jobs.create": true}
DockerImageFormats: {"v2": true}
Crunch1:
Enable: true
CrunchJobWrapper: none
CrunchJobUser: crunch
CrunchRefreshTrigger: /tmp/crunch_refresh_trigger
DefaultDockerImage: false
NodeProfiles:
# Key is a profile name; can be specified on service prog command line, defaults to $(hostname)
keep:
# Don’t run other services automatically -- only specified ones
Default: {Disable: true}
Keepstore: {Listen: ":25107"}
apiserver:
Default: {Disable: true}
RailsAPI: {Listen: ":9000", TLS: true}
Controller: {Listen: ":9100"}
Websocket: {Listen: ":9101"}
Health: {Listen: ":9199"}
keep:
Default: {Disable: true}
KeepProxy: {Listen: ":9102"}
KeepWeb: {Listen: ":9103"}
*:
# This section used for a node whose profile name is not listed above
Default: {Disable: false} # (this is the default behavior)
Volumes:
xyzzy-keep-0:
Type: s3
Region: us-east
Bucket: xyzzy-keep-0
# [rest of keepstore volume config goes here]
WebRoutes:
# “default” means route according to method/host/path (e.g., if host is a login shell, route there)
xyzzy.arvadosapi.com: default
# “collections” means always route to keep-web
collections.xyzzy.arvadosapi.com: collections
# leading * is a wildcard (longest match wins)
"*--collections.xyzzy.arvadosapi.com": collections
cloud.curoverse.com: workbench
workbench.xyzzy.arvadosapi.com: workbench
"*.xyzzy.arvadosapi.com": default
InstanceTypes:
m4.large:
VCPUs: 2
RAM: 8000000000
Scratch: 31000000000
Price: 0.1
m4.large-1t:
# same instance type as m4.large but our scripts attach more scratch
ProviderType: m4.large
VCPUs: 2
RAM: 8000000000
Scratch: 999000000000
Price: 0.12
m4.xlarge:
VCPUs: 4
RAM: 16000000000
Scratch: 78000000000
Price: 0.2
m4.8xlarge:
VCPUs: 40
RAM: 160000000000
Scratch: 156000000000
Price: 2
m4.16xlarge:
VCPUs: 64
RAM: 256000000000
Scratch: 310000000000
Price: 3.2
c4.large:
VCPUs: 2
RAM: 3750000000
Price: 0.1
c4.8xlarge:
VCPUs: 36
RAM: 60000000000
Price: 1.591
RemoteClusters:
xrrrr:
Host: xrrrr.arvadosapi.com
Proxy: true # proxy requests to xrrrr on behalf of our clients
AuthProvider: true # users authenticated by xrrrr can use our cluster
</code></pre>