Migrating from arvados-node-manager to arvados-dispatch-cloud » History » Version 1

Tom Clegg, 02/07/2019 08:07 PM

1 1 Tom Clegg
h1. Migrating from arvados-node-manager to crunch-dispatch-cloud
2 1 Tom Clegg
3 1 Tom Clegg
{{toc}}
4 1 Tom Clegg
5 1 Tom Clegg
h2. Choose a node
6 1 Tom Clegg
7 1 Tom Clegg
The dispatch service can run on any host that can connect to the Arvados API service, the cloud provider's API, and the SSH service on cloud VMs. In the following example it runs on the same node as the API server and controller.
8 1 Tom Clegg
9 1 Tom Clegg
h2. Update cluster configuration file
10 1 Tom Clegg
11 1 Tom Clegg
In @/etc/arvados/config.yml@, add configuration items for the dispatch service.
12 1 Tom Clegg
13 1 Tom Clegg
<pre><code class="yaml">
14 1 Tom Clegg
Clusters:
15 1 Tom Clegg
  uuid_prefix:
16 1 Tom Clegg
    CloudVMs:
17 1 Tom Clegg
      BootProbeCommand: "mount | grep /mnt/scratch"
18 1 Tom Clegg
      SSHPort: "2222"
19 1 Tom Clegg
      SyncInterval: 1m
20 1 Tom Clegg
      TimeoutIdle: 2m
21 1 Tom Clegg
      TimeoutBooting: 10m
22 1 Tom Clegg
      TimeoutProbe: 5m
23 1 Tom Clegg
      TimeoutShutdown: 30s
24 1 Tom Clegg
      ImageID: "image-12345678"
25 1 Tom Clegg
      Driver: Azure
26 1 Tom Clegg
      DriverParameters:
27 1 Tom Clegg
        ...
28 1 Tom Clegg
    Dispatch:
29 1 Tom Clegg
      PrivateKey: "..."
30 1 Tom Clegg
      StaleLockTimeout: 1m
31 1 Tom Clegg
      PollInterval: 10s
32 1 Tom Clegg
      ProbeInterval: 10s
33 1 Tom Clegg
      MaxProbesPerSecond: 10
34 1 Tom Clegg
    InstanceTypes:
35 1 Tom Clegg
      x1lg:
36 1 Tom Clegg
        ProviderType: x1.large
37 1 Tom Clegg
        VCPUs: 16
38 1 Tom Clegg
        RAM: 128G
39 1 Tom Clegg
        Scratch: 128G
40 1 Tom Clegg
        Price: 1.23
41 1 Tom Clegg
    ManagementToken: "example-secret-management-token"
42 1 Tom Clegg
    NodeProfiles:
43 1 Tom Clegg
      apiserver:                       # references ARVADOS_NODE_PROFILE in environment file (see below).
44 1 Tom Clegg
        arvados-dispatch-cloud:
45 1 Tom Clegg
          Listen: ":9005"
46 1 Tom Clegg
</code></pre>
47 1 Tom Clegg
48 1 Tom Clegg
Create the host configuration file @/etc/arvados/environment@.
49 1 Tom Clegg
50 1 Tom Clegg
<pre>
51 1 Tom Clegg
ARVADOS_NODE_PROFILE=apiserver
52 1 Tom Clegg
</pre>
53 1 Tom Clegg
54 1 Tom Clegg
h2. Stop crunch-dispatch-slurm
55 1 Tom Clegg
56 1 Tom Clegg
Stop and disable the crunch-dispatch-slurm service, and uninstall the package to make sure it doesn't start after the next reboot/upgrade.
57 1 Tom Clegg
58 1 Tom Clegg
<pre>
59 1 Tom Clegg
# systemctl stop crunch-dispatch-slurm
60 1 Tom Clegg
# systemctl disable crunch-dispatch-slurm
61 1 Tom Clegg
# apt-get remove crunch-dispatch-slurm
62 1 Tom Clegg
</pre>
63 1 Tom Clegg
64 1 Tom Clegg
Containers that have already been locked and submitted to SLURM will make their way through the SLURM queue, but newly queued containers will be left for crunch-dispatch-cloud to run.
65 1 Tom Clegg
66 1 Tom Clegg
h2. Install crunch-dispatch-cloud
67 1 Tom Clegg
68 1 Tom Clegg
<pre>
69 1 Tom Clegg
# apt-get install crunch-dispatch-cloud
70 1 Tom Clegg
</pre>
71 1 Tom Clegg
72 1 Tom Clegg
h2. Verify the service is running
73 1 Tom Clegg
74 1 Tom Clegg
<pre>
75 1 Tom Clegg
$ token="example-secret-management-token"
76 1 Tom Clegg
$ curl -H "Authorization: Bearer $token" http://localhost:9005/metrics
77 1 Tom Clegg
</pre>
78 1 Tom Clegg
79 1 Tom Clegg
h2. Verify the service is functional