Cluster configuration¶

We are (2018) consolidating configuration from per-microservice yaml/json/ini files into a single cluster configuration document that is used by all components.

• Long term: system nodes automatically keep their configs synchronized (using something like consul).
• Short term: sysadmin uses tools like puppet and terraform to ensure /etc/arvados/config.yml is identical on all system nodes.
• Hosts without config files (e.g., hosts outside the cluster) can retrieve the config document from the API server.

Discovery document¶

Previously, we copied selected config values from the API server config into the API discovery document so clients could see them. When clients can get the configuration document itself, this won't be needed. The discovery document should advertise APIs provided by the server, not cluster configuration.

Secrets¶

Secrets like BlobSigningKey can be given literally in the config file (convenient for dev/test, consul-template, etc) or indirectly using a secret backend. Anticipated backends:

• BlobSigningKey: foobar ⇒ the secret is literally foobar
• BlobSigningKey: "vault:foobar" ⇒ the secret can be obtained from vault using the vault key "foobar"
• BlobSigningKey: "file:/foobar" ⇒ the secret can be read from the local file /foobar
• BlobSigningKey: "env:FOOBAR" ⇒ the secret can be read from the environment variable FOOBAR

Example config file¶

(Format not yet frozen!)

Clusters:
  xyzzy:
    BlobSigningKey: ungu355able
    BlobSignatureTTL: 172800
    SessionKey: 186005aa54cab1ca95a3738e6e954e0a35a96d3d13a8ea541f4156e8d067b4f3
    PostgreSQL:
      ConnectionPool: 32 # max concurrent connections per arvados server daemon
      Connection:
        # All parameters here are passed to the PG client library in a connection string;
        # see https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS
        Host: localhost
        Port: 5432
        User: arvados
        Password: s3cr3t
        DBName: arvados_production
        client_encoding: utf8
        fallback_application_name: arvados
    HTTPRequestTimeout: 5m
    Defaults:
      CollectionReplication: 2
      TrashLifetime: 2w
    UserActivation:
      ActivateNewUsers: true
      AutoAdminUser: root@example.com
      UserProfileNotificationAddress: notify@example.com
      NewUserNotificationRecipients: {}
      NewInactiveUserNotificationRecipients: {}
    Limits:
      MaxRequestLogParamsSize: 2KB
      MaxRequestSize: 128MiB
      MaxIndexDatabaseRead: 128MiB
      MaxItemsPerResponse: 1000
    LoggingLevel:
      Default: INFO
      apiclient: WARNING
      googleapiclient: WARNING
    NodeManager:
      Dispatcher: slurm
      PollTime: 10s
      BootFailAfter: 1200s
      Arvados:
        Token: (redacted)
        Timeout: 20s
      Cloud:
        Provider: AWS
        Region: us-east-1
        Timeout: 20s
        ShutdownWindows: 21, 999999
        NodeCreate:
          PingHost: xyzzy.arvadosapi.com
          ExKeyname: compute
          ImageID: ami-0a01b48b88d14541e
          SubnetID: subnet-24f5ae62
          SecurityGroups: sg-3ec53e2a
        NodeList:
          InstanceStateName: running
          Tags:
            arvados-class: dynamic-compute
            cluster: xyzzy
    AuditLogs:
      MaxAge: 2w
      DeleteBatchSize: 100000
      UnloggedAttributes: {} # example: {"manifest_text": true}
    ContainerLogStream:
      BatchSize: 4KiB
      BatchTime: 1s
      ThrottlePeriod: 1m
      ThrottleThresholdSize: 64KiB
      ThrottleThresholdLines: 1024
      TruncateSize: 64MiB
      PartialLineThrottlePeriod: 5s
    Timers:
      TrashSweepInterval: 60s
    Scaling:
      MaxComputeNodes: 64
      EnablePreemptibleInstances: false
    DisableAPIMethods: {} # example: {"jobs.create": true}
    DockerImageFormats: {"v2": true}
    Crunch1:
      Enable: true
      CrunchJobWrapper: none
      CrunchJobUser: crunch
      CrunchRefreshTrigger: /tmp/crunch_refresh_trigger
      DefaultDockerImage: false
    NodeProfiles:
      # Key is a profile name; can be specified on service prog command line, defaults to $(hostname)
      keep:
        # Don’t run other services automatically -- only specified ones
        Default: {Disable: true}
        Keepstore: {Listen: ":25107"}
      apiserver:
        Default: {Disable: true}
        RailsAPI: {Listen: ":9000", TLS: true}
        Controller: {Listen: ":9100"}
        Websocket: {Listen: ":9101"}
        Health: {Listen: ":9199"}
      keep:
        Default: {Disable: true}
        KeepProxy: {Listen: ":9102"}
        KeepWeb: {Listen: ":9103"}
      *:
        # This section used for a node whose profile name is not listed above
        Default: {Disable: false} # (this is the default behavior)
    Volumes:
      xyzzy-keep-0:
        Type: s3
        Region: us-east
        Bucket: xyzzy-keep-0
        # [rest of keepstore volume config goes here]
    Providers:
      AWS:
        # [credentials and stuff go here]
        us-east-1:
          Compute:
            Key: ABCDEF
            Secret: abcdef123456abcdef123456abcdef123456
          Storage:
            Key: 987ABC
            Secret: 456deadbeef123456deadbeef123456deadbeef123
    WebRoutes:
      # “default” means route according to method/host/path (e.g., if host is a login shell, route there)
      xyzzy.arvadosapi.com: default
      # “collections” means always route to keep-web
      collections.xyzzy.arvadosapi.com: collections
      # leading * is a wildcard (longest match wins)
      "*--collections.xyzzy.arvadosapi.com": collections
      cloud.curoverse.com: workbench
      workbench.xyzzy.arvadosapi.com: workbench
      "*.xyzzy.arvadosapi.com": default
    InstanceTypes:
      m4.large:
        VCPUs: 2
        RAM: 8000000000
        Scratch: 31000000000
        Price: 0.1
      m4.large-1t:
        # same instance type as m4.large but our scripts attach more scratch
        ProviderType: m4.large
        VCPUs: 2
        RAM: 8000000000
        Scratch: 999000000000
        Price: 0.12
      m4.xlarge:
        VCPUs: 4
        RAM: 16000000000
        Scratch: 78000000000
        Price: 0.2
      m4.8xlarge:
        VCPUs: 40
        RAM: 160000000000
        Scratch: 156000000000
        Price: 2
      m4.16xlarge:
        VCPUs: 64
        RAM: 256000000000
        Scratch: 310000000000
        Price: 3.2
      c4.large:
        VCPUs: 2
        RAM: 3750000000
        Price: 0.1
      c4.8xlarge:
        VCPUs: 36
        RAM: 60000000000
        Price: 1.591
    RemoteClusters:
      xrrrr:
        Host: xrrrr.arvadosapi.com
        Proxy: true        # proxy requests to xrrrr on behalf of our clients
        AuthProvider: true # users authenticated by xrrrr can use our cluster