Story #14931

[arvados-dispatch-cloud] Configurable instance tags

Added by Tom Clegg almost 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
-
Target version:
2019-06-05 Sprint
Start date:
05/31/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0
Release:
Arvados 2.0 - Early 2020
Release relationship:
Auto

Description

Two related features would help make arvados-dispatch-cloud play nicer with other ways instance tags are used in the same cloud account:
  1. Admin can supply some tags that get set on all instances created by arvados-dispatch-cloud (e.g., {"myOrg-instance-class": "arvados", "admin-contact": ""})
  2. Admin can control the tag prefix (currently a prefix is hard-coded by each driver, but this should be moved out to the dispatcher, and the driver should just do exactly what it's told)

Instance-specific resources (like NICs in Azure) should also be tagged, using the same tags as the instance.

Shared resources (like key pairs in EC2) should be tagged with the preconfigured tags, but not the per-instance tags. (Implementation: the driver doesn't know which tags are instance-specific, so the set of tags to use for shared resources should be supplied explicitly during driver initialization.)

Tag keys and values are case sensitive.

Subtasks

Task #15263: Review 14931-custom-tagsIn ProgressPeter Amstutz

Related issues

Related to Arvados - Feature #14291: [crunch-dispatch-cloud] AWS driverResolved02/28/2019

Related to Arvados - Story #13908: [Epic] Replace SLURM for cloud job scheduling/dispatchingResolved

Related to Arvados - Feature #15063: [a-d-c] Assign names to EC2 instancesDuplicate

Has duplicate Arvados - Story #15075: [a-d-c] Support for extra tags on resources created by arvados Duplicate

Associated revisions

Revision c792e499
Added by Tom Clegg over 1 year ago

Merge branch '14931-custom-tags'

closes #14931

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg almost 2 years ago

#2 Updated by Tom Clegg almost 2 years ago

  • Related to Story #13908: [Epic] Replace SLURM for cloud job scheduling/dispatching added

#3 Updated by Tom Morris almost 2 years ago

  • Tracker changed from Bug to Story

#4 Updated by Tom Morris almost 2 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 1.0

#5 Updated by Tom Morris almost 2 years ago

  • Related to Story #15075: [a-d-c] Support for extra tags on resources created by arvados added

#6 Updated by Tom Clegg almost 2 years ago

  • Description updated (diff)

#7 Updated by Tom Clegg almost 2 years ago

  • Description updated (diff)

#8 Updated by Tom Clegg almost 2 years ago

  • Has duplicate Story #15075: [a-d-c] Support for extra tags on resources created by arvados added

#9 Updated by Tom Clegg almost 2 years ago

  • Related to deleted (Story #15075: [a-d-c] Support for extra tags on resources created by arvados )

#10 Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress
  • Assigned To set to Tom Clegg
  • Target version changed from Arvados Future Sprints to 2019-06-05 Sprint

#11 Updated by Tom Clegg over 1 year ago

  • Related to Feature #15063: [a-d-c] Assign names to EC2 instances added

#13 Updated by Peter Amstutz over 1 year ago

I think the default value of TagKeyPrefix in config.yml.default should be "arvados-" instead of blank.

It looks like you can set a static value for a tag like "name" (so you can have a bunch of instances called "arvados compute" which is better than being blank) but do we want a way to set the name based on the Arvados instance id? I'm thinking of the case where the admin needs to correlate an instance on the native cloud dashboard with cloud-dispatch logs. Perhaps the tag value could be interpreted as a format string?

#14 Updated by Tom Clegg over 1 year ago

Peter Amstutz wrote:

I think the default value of TagKeyPrefix in config.yml.default should be "arvados-" instead of blank.

How about Arvados, so we get ArvadosInstanceType etc.?

It looks like you can set a static value for a tag like "name" (so you can have a bunch of instances called "arvados compute" which is better than being blank) but do we want a way to set the name based on the Arvados instance id? I'm thinking of the case where the admin needs to correlate an instance on the native cloud dashboard with cloud-dispatch logs. Perhaps the tag value could be interpreted as a format string?

Dispatch-cloud uses the provider-assigned instance ID instead of inventing its own for logs, status, etc., so I don't think an additional cross-reference field is needed.

Also, that provider-assigned instance ID isn't known yet when we supply the initial tags for a new instance, so we'd have to start with a fake value and then change it on next sync, which would create edge cases for anyone trying to use it.

#15 Updated by Peter Amstutz over 1 year ago

Tom Clegg wrote:

Peter Amstutz wrote:

I think the default value of TagKeyPrefix in config.yml.default should be "arvados-" instead of blank.

How about Arvados, so we get ArvadosInstanceType etc.?

Sounds good to me.

It looks like you can set a static value for a tag like "name" (so you can have a bunch of instances called "arvados compute" which is better than being blank) but do we want a way to set the name based on the Arvados instance id? I'm thinking of the case where the admin needs to correlate an instance on the native cloud dashboard with cloud-dispatch logs. Perhaps the tag value could be interpreted as a format string?

Dispatch-cloud uses the provider-assigned instance ID instead of inventing its own for logs, status, etc., so I don't think an additional cross-reference field is needed.

Also, that provider-assigned instance ID isn't known yet when we supply the initial tags for a new instance, so we'd have to start with a fake value and then change it on next sync, which would create edge cases for anyone trying to use it.

Sorry about that, I was thinking of Azure where the instance ids are made up by the driver. You're right, so long as the instance ids in the logs match up with the instance ids on the cloud's dashboard, there is nothing to do here.

Please merge after updating the TagKeyPrefix default.

#16 Updated by Tom Clegg over 1 year ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#17 Updated by Peter Amstutz about 1 year ago

  • Release set to 22

Also available in: Atom PDF