Documentation
¶
Overview ¶
Package aws implements driver
Index ¶
- Constants
- type Config
- type DedicatedPoolRecord
- type Driver
- func (d *Driver) Allocate(def types.LabelDefinition, metadata map[string]any) (*types.ApplicationResource, error)
- func (d *Driver) AvailableCapacity(_ types.Resources, def types.LabelDefinition) int64
- func (d *Driver) Deallocate(res *types.ApplicationResource) error
- func (d *Driver) GetTask(name, options string) provider.DriverTask
- func (*Driver) IsRemote() bool
- func (d *Driver) Name() string
- func (d *Driver) Prepare(config []byte) error
- func (d *Driver) SetName(name string)
- func (d *Driver) Status(res *types.ApplicationResource) (string, error)
- func (d *Driver) ValidateDefinition(def types.LabelDefinition) error
- type Factory
- type Options
- type TaskImage
- type TaskSnapshot
Constants ¶
const HostReserved = "reserved"
HostReserved - custom status to set in the host for simplifying parallel ops in between the updates
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
Region string `json:"region"` // AWS Region to connect to
KeyID string `json:"key_id"` // AWS AMI Key ID
SecretKey string `json:"secret_key"` // AWS AMI Secret Key
// Optional
AccountIDs []string `json:"account_ids"` // AWS Trusted account IDs to filter vpc, subnet, sg, images, snapshots...
InstanceTags map[string]string `json:"instance_tags"` // AWS Instance tags to use when this node provision them
InstanceKey string `json:"instance_key"` // AWS Instance Key Pair name to use while creating of the instance
// Manage the AWS dedicated hosts to keep them busy and deallocate when not needed
// Key of the map is name of the pool - will be used for identification of the pool
DedicatedPool map[string]DedicatedPoolRecord `json:"dedicated_pool"`
// Various options to not hardcode the important numbers
SnapshotCreateWait util.Duration `json:"snapshot_create_wait"` // Maximum wait time for snapshot availability (create), default: 2h
ImageCreateWait util.Duration `json:"image_create_wait"` // Maximum wait time for image availability (create/copy), default: 2h
}
Config - node driver configuration
type DedicatedPoolRecord ¶
type DedicatedPoolRecord struct {
Type string `json:"type"` // Instance type handled by the dedicated hosts pool (example: "mac2.metal")
Zones []string `json:"zones"` // Where to allocate the dedicated host (example: ["us-west-2a", "us-west-2c"])
Max uint `json:"max"` // Maximum dedicated hosts to allocate (they sometimes can handle more than 1 capacity slot)
// AWS has a bug in it's API - when you getting the dedicated hosts availability it can say the
// host is become available, but in fact it's not. Particularly we see that with mac machines
// when they are returning after scrubbing. So this delay will add the requested delay between
// previous state of the dedicated host and available state, so the allocations will not fail.
PendingToAvailableDelay util.Duration `json:"pending_to_available_delay"`
// Specifies when the dedicated host can be released after allocation. By default for mac type
// it's set to [24h] but you can set to half a year or more to keep the host in your pool as
// long as you need. If you want to set it to lower value for mac, then 24h - please consult
// the AWS docs regarding that.
//
// [24h]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-considerations
ReleaseDelay util.Duration `json:"release_delay"`
// Is a special optimization for the Mac dedicated hosts to send them in [scrubbing process] to
// save money when we can't release the host due to Apple's license of [24 hours] min limit.
//
// Details:
//
// Apple forces AWS and any of their customers to keep the Mac dedicated hosts allocated for at
// least [24 hours]. So after allocation you have no way to release the dedicated host even if
// you don't need it. This makes the mac hosts very pricey for any kind of dynamic allocation.
// In order to workaround this issue - Aquarium implements optimization to keep the Mac hosts
// busy with [scrubbing process], which is triggered after the instance stop or termination and
// puts Mac host in pending state for 1-2hr. That's the downside of optimization, because you
// not be able to use the machine until it will become available again.
//
// That's why this ScrubbingDelay config exists - we need to give Mac host some time to give
// the workload a chance to utilize the host. If it will not be utilized in this duration - the
// manager will start the scrubbing process. When the host become old enough - the manager will
// release it to clean up space for new fresh mac in the roster.
//
// * When this option is unset or 0 - no optimization is enabled.
// * When it's set - then it's a duration to stay idle and then allocate and terminate empty
// instance to trigger scrubbing.
//
// Current implementation is attached to state update, which could be API consuming, so this
// duration should be >= 1 min, otherwise API requests will be too often.
//
// [24 hours]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-considerations
// [scrubbing process]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mac-instance-stop.html
ScrubbingDelay util.Duration `json:"scrubbing_delay"`
}
DedicatedPoolRecord stores the configuration of AWS dedicated pool of particular type to manage aws ec2 allocate-hosts --availability-zone "us-west-2c" --auto-placement "on" --host-recovery "off" --host-maintenance "off" --quantity 1 --instance-type "mac2.metal"
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver implements provider.Driver interface
func (*Driver) Allocate ¶
func (d *Driver) Allocate(def types.LabelDefinition, metadata map[string]any) (*types.ApplicationResource, error)
Allocate Instance with provided image
It selects the AMI and run instance Uses metadata to fill EC2 instance userdata
func (*Driver) AvailableCapacity ¶
AvailableCapacity allows Fish to ask the driver about it's capacity (free slots) of a specific definition
func (*Driver) Deallocate ¶
func (d *Driver) Deallocate(res *types.ApplicationResource) error
Deallocate the resource
func (*Driver) GetTask ¶
func (d *Driver) GetTask(name, options string) provider.DriverTask
GetTask returns task struct by name
func (*Driver) IsRemote ¶
IsRemote needed to detect the out-of-node resources managed by this driver
func (*Driver) Status ¶
func (d *Driver) Status(res *types.ApplicationResource) (string, error)
Status shows status of the resource
func (*Driver) ValidateDefinition ¶
func (d *Driver) ValidateDefinition(def types.LabelDefinition) error
ValidateDefinition checks LabelDefinition is ok
type Options ¶
type Options struct {
Image string `json:"image"` // ID/Name of the image you want to use (name that contains * is usually a bad idea for reproducibility)
InstanceType string `json:"instance_type"` // Type of the instance from aws available list
SecurityGroup string `json:"security_group"` // ID/Name of the security group to use for the instance
Tags map[string]string `json:"tags"` // Tags to add during instance creation
EncryptKey string `json:"encrypt_key"` // Use specific encryption key for the new disks
Pool string `json:"pool"` // Use machine from dedicated pool, otherwise will try to use one with auto-placement
UserDataFormat string `json:"userdata_format"` // If not empty - will store the resource metadata to userdata in defined format
UserDataPrefix string `json:"userdata_prefix"` // Optional if need to add custom prefix to the metadata key during formatting
// TaskImage options
TaskImageName string `json:"task_image_name"` // Create new image with defined name + "-DATE.TIME" suffix
TaskImageEncryptKey string `json:"task_image_encrypt_key"` // KMS Key ID or Alias in format "alias/<name>" if need to re-encrypt the newly created AMI snapshots
}
Options for label definition
Example:
image: ami-abcdef123456 instance_type: c6a.4xlarge security_group: sg-abcdef123456 tags: somekey: somevalue
type TaskImage ¶
type TaskImage struct {
*types.ApplicationTask `json:"-"` // Info about the requested task
*types.LabelDefinition `json:"-"` // Info about the used label definition
*types.ApplicationResource `json:"-"` // Info about the processed resource
Full bool `json:"full"` // Make full (all disks including connected disks), or just the root OS disk image
// contains filtered or unexported fields
}
TaskImage stores the task data
func (*TaskImage) Clone ¶
func (t *TaskImage) Clone() provider.DriverTask
Clone makes a copy of the initial task to execute
func (*TaskImage) Execute ¶
Execute - Image task could be executed during ALLOCATED & DEALLOCATE ApplicationStatus
func (*TaskImage) SetInfo ¶
func (t *TaskImage) SetInfo(task *types.ApplicationTask, def *types.LabelDefinition, res *types.ApplicationResource)
SetInfo defines information of the environment
type TaskSnapshot ¶
type TaskSnapshot struct {
*types.ApplicationTask `json:"-"` // Info about the requested task
*types.LabelDefinition `json:"-"` // Info about the used label definition
*types.ApplicationResource `json:"-"` // Info about the processed resource
Full bool `json:"full"` // Make full (all disks including OS image), or just the additional disks snapshot
// contains filtered or unexported fields
}
TaskSnapshot stores the task data
func (*TaskSnapshot) Clone ¶
func (t *TaskSnapshot) Clone() provider.DriverTask
Clone makes a copy of the initial task to execute
func (*TaskSnapshot) Execute ¶
func (t *TaskSnapshot) Execute() (result []byte, err error)
Execute - Snapshot task could be executed during ALLOCATED & DEALLOCATE ApplicationStatus
func (*TaskSnapshot) SetInfo ¶
func (t *TaskSnapshot) SetInfo(task *types.ApplicationTask, def *types.LabelDefinition, res *types.ApplicationResource)
SetInfo defines information of the environment