dax

package
v3.26.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2022 License: Apache-2.0 Imports: 15 Imported by: 0

README

DAX

DAX encapsulates anything which covers all of the services which make up the "disaggregation of storage and compute" project. Initially, this will include integration tests which pull in things like Metadata Services (MDS), including the Controller, as well as FeatureBase and IDK-based ingesters.

Setting up the tests

The DAX test currently requires docker images for: featurebase and datagen.

If at any point you run into problems with go mod failing to reference a private repo, make sure that you have gitlab.com/molecula in your GOPRIVATE environment variable.

Note that during the docker image build step, go mod vendor is run, which creates a vendor directory in the root directory, and copies that to docker during the build stage. Just be aware of this; you may want to remove that vendor directory after you're done building docker images.

Possible Configuruation

The following were relevent when the dax code was in a separate repository. These may no longer be relevant.

I needed to but this in my ~/.profile file:

export GOPRIVATE=github.com/molecula,gitlab.com/molecula

And this in my ~/.gitconfig

[url "ssh://git@github.com/"]
	insteadOf = https://github.com/
[url "ssh://git@gitlab.com/"]
	insteadOf = https://gitlab.com/

Then make docker ran successfully.

Build the FeatureBase docker image
  • Check out the dax branch of the featurebase repository.
  • Run make docker-image-featurebase to build the docker image
  • You should now have an image in docker named dax/featurebase with the tag latest.
Build the Datagen docker image
  • cd <featurebase_repo_root>/idk
  • Run make docker-image-datagen to build the docker image
  • You should now have an image in docker named dax/datagen with the tag latest.

Running the tests

  • Check out the dax branch of the featurebase repository.
  • Change into the dax directory: cd dax
  • Run make test-integration.

Documentation

Overview

Package dax defines DAX domain level types.

Index

Constants

View Source 
const (
	ServicePrefixComputer    = "computer"
	ServicePrefixMDS         = "mds"
	ServicePrefixQueryer     = "queryer"
	ServicePrefixSnapshotter = "snapshotter"
	ServicePrefixWriteLogger = "writelogger"
)

ServicePrefixes are used as the service prefix value in http handlers.

View Source 
const (
	ErrTableIDExists         errors.Code = "TableIDExists"
	ErrTableKeyExists        errors.Code = "TableKeyExists"
	ErrTableNameExists       errors.Code = "TableNameExists"
	ErrTableIDDoesNotExist   errors.Code = "TableIDDoesNotExist"
	ErrTableKeyDoesNotExist  errors.Code = "TableKeyDoesNotExist"
	ErrTableNameDoesNotExist errors.Code = "TableNameDoesNotExist"

	ErrFieldExists       errors.Code = "FieldExists"
	ErrFieldDoesNotExist errors.Code = "FieldDoesNotExist"

	ErrUnimplemented errors.Code = "Unimplemented"
)
View Source 
const (
	BaseTypeBool      = "bool"      //
	BaseTypeDecimal   = "decimal"   //
	BaseTypeID        = "id"        // non-keyed mutex
	BaseTypeIDSet     = "idset"     // non-keyed set
	BaseTypeInt       = "int"       //
	BaseTypeString    = "string"    // keyed mutex
	BaseTypeStringSet = "stringset" // keyed set
	BaseTypeTimestamp = "timestamp" //

	DefaultPartitionN = 256

	PrimaryKeyFieldName = FieldName("_id")
)

Base types.

View Source 
const (
	ErrNodeDoesNotExist errors.Code = "NodeDoesNotExist"
)
View Source 
const PrefixTable = "tbl"

PrefixTable is used as a prefix to TableKey strings because FeatureBase indexes must start with an alpha (a-z) character. Because the string representation of a uuid (i.e. the OrganizationID value) can start with a numeric value, we can't have OrganizationId (or any of the other ID values which make up the TableKey) be at the beginning of the TableKey.

View Source 
const TableKeyDelimiter = "__"

TableKeyDelimiter is used to delimit the qualifer elements in the TableKey. While it might make more sense to use a pipe ("|") here, we instead use a double underscore because underscore is one of the few characters allowed by the FeatureBase index name restrictions, and we double it in a lame attempt to distinquish it from FeatureBase index names which contain a single underscore.

Variables

This section is empty.

Functions

func NewErrFieldDoesNotExist 

func NewErrFieldDoesNotExist(fieldName FieldName) error

func NewErrFieldExists 

func NewErrFieldExists(fieldName FieldName) error

func NewErrNodeDoesNotExist 

func NewErrNodeDoesNotExist(addr Address) error

func NewErrTableIDDoesNotExist 

func NewErrTableIDDoesNotExist(qtid QualifiedTableID) error

func NewErrTableIDExists 

func NewErrTableIDExists(qtid QualifiedTableID) error

func NewErrTableKeyDoesNotExist 

func NewErrTableKeyDoesNotExist(tkey TableKey) error

func NewErrTableKeyExists 

func NewErrTableKeyExists(tkey TableKey) error

func NewErrTableNameDoesNotExist 

func NewErrTableNameDoesNotExist(tableName TableName) error

func NewErrTableNameExists 

func NewErrTableNameExists(tableName TableName) error

func NewNopVersionStore 

func NewNopVersionStore() *nopVersionStore

NewNopVersionStore returns a new no-op instance of VersionStore.

Types

type Address 

type Address string

Address is a string of the form [scheme]://[host]:[port]/path

func (Address) Host 

func (a Address) Host() string

Host returns the [host] portion of the Address.

func (Address) HostPort 

func (a Address) HostPort() string

HostPort returns the [host]:[port] portion of the Address; in other words, the Address stripped of any scheme and path.

func (Address) OverrideScheme 

func (a Address) OverrideScheme(scheme string) string

OverrideScheme overrides Address's current scheme with the one provided. If an empty scheme is provided, OverrideScheme will return just the host:port/path.

func (Address) Path 

func (a Address) Path() string

Path returns the path portion of the Address.

func (Address) Port 

func (a Address) Port() uint16

Port returns the [port] portion of the Address. If the port values is invalid or does not exist, the returned value will default to 0.

func (Address) Scheme 

func (a Address) Scheme() string

Scheme returns the [scheme] portion of the Address. This may be an empty string if Address does not contain a scheme.

func (Address) String 

func (a Address) String() string

String returns the Address as a string type.

func (Address) WithScheme 

func (a Address) WithScheme(dflt string) string

WithScheme ensures that the string returned contains the scheme portion of a URL. Because an Address may not have a scheme (for example, it could be just "host:80"), this method can be applied to an address when it needs to be used as a URL. If the address's existing scheme is blank, the default scheme provided will be used. If address is blank, the default scheme will not be added; i.e. address will remain blank.

type AddressManager 

type AddressManager interface {
	AddAddresses(context.Context, ...Address) error
	RemoveAddresses(context.Context, ...Address) error
}

AddressManager is an interface for any service which needs to maintain a list of addresses, and receive add/remove address requests from other services.

type AssignedNode 

type AssignedNode struct {
	Address Address `json:"address"`
	Role    Role    `json:"role"`
}

AssignedNode is used in API responses.

type BaseType 

type BaseType string

BaseType is a typed string used for field types.

func BaseTypeFromString 

func BaseTypeFromString(s string) (BaseType, error)

BaseTypeFromString converts a string to one of the defined BaseTypes. If the string does not match a BaseType, then an error is returned.

type ComputeRole 

type ComputeRole struct {
	TableKey TableKey        `json:"table-key"`
	Shards   VersionedShards `json:"shards"`
}

ComputeRole is a role specific to compute nodes.

func (*ComputeRole) Type 

func (cr *ComputeRole) Type() RoleType

Type returns the type for ComputeRole. This is mainly to impolement the Role interface.

type ComputerService 

type ComputerService interface {
	MultiService

	SetMDS(Address) error
}

type DatabaseID 

type DatabaseID string

DatabaseID is the unique database identifier, currently generated by the Control Plane in a FeatureBase cloud implementation. In that implementation, its value is a uuid as a string, but there's nothing enforcing that; the value could be any string.

type Directive 

type Directive struct {
	Address Address `json:"address"`

	// Method describes how the compute node should handle the Directive. See
	// the different constants of type DirectiveMethod for how this value is
	// handled.
	Method DirectiveMethod `json:"method"`

	Tables []*QualifiedTable `json:"schema"`

	ComputeRoles   []ComputeRole   `json:"compute-roles"`
	TranslateRoles []TranslateRole `json:"translate-roles"`

	Version uint64 `json:"version"`
}

Directive contains the instructions, sent from MDS, which a compute node is to follow. A Directive is typically JSON-encoded and POSTed to a compute node's `/directive` endpoint.

func (*Directive) ComputeShards 

func (d *Directive) ComputeShards(tbl TableKey) VersionedShards

ComputeShards returns the list of shards, for the given table, for which this compute node is responsible. It assumes that the Directive does not contain more than one ComputeRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) ComputeShardsMap 

func (d *Directive) ComputeShardsMap() map[TableKey]VersionedShards

ComputeShardsMap returns a map of table to shards. It assumes that the Directive does not contain more than one ComputeRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) IsEmpty 

func (d *Directive) IsEmpty() bool

IsEmpty tells whether a directive is assigning actual responsibilty to a node or not. If the directive does not assign responsibility for any shard or partition then it is considered empty. This is used to determine whether we can ignore an error received from applying this directive (an empty directive is often sent to a node which is already down).

func (*Directive) Table 

func (d *Directive) Table(qtid QualifiedTableID) (*QualifiedTable, error)

Table returns the ID'd table from the Directive's Tables list. If it's not found, it returns nil and a non-nil error. A nil error guarantees that the returned table is non-nil.

func (*Directive) TranslateFieldsMap 

func (d *Directive) TranslateFieldsMap() map[TableKey]VersionedFields

TranslateFieldsMap returns a map of table to fields. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of FieldValues.

func (*Directive) TranslatePartitions 

func (d *Directive) TranslatePartitions(tbl TableKey) VersionedPartitions

TranslatePartitions returns the list of partitions, for the given table, for which this translate node is responsible. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) TranslatePartitionsMap 

func (d *Directive) TranslatePartitionsMap() map[TableKey]VersionedPartitions

TranslatePartitionsMap returns a map of table to partitions. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of Partitions.

type DirectiveMethod 

type DirectiveMethod string

DirectiveMethod is used to tell the compute node how it should handle the Directive. 

const (
	// DirectiveMethodDiff tells the compute node to diff the Directive with its
	// local, cached Directive and only apply the differences.
	DirectiveMethodDiff DirectiveMethod = "diff"

	// DirectiveMethodReset tells the compute node to delete all of its existing
	// data before applying the directive.
	DirectiveMethodReset DirectiveMethod = "reset"

	// DirectiveMethodSnapshot tells the compute node that the incoming
	// Directive should only contain data version updates related to a snapshot
	// request.
	DirectiveMethodSnapshot DirectiveMethod = "snapshot"
)

type DirectiveVersion 

type DirectiveVersion interface {
	Increment(ctx context.Context, delta uint64) (uint64, error)
}

type Directives 

type Directives []*Directive

Directives is a sortable slice of Directive.

func (Directives) Len 

func (d Directives) Len() int

func (Directives) Less 

func (d Directives) Less(i, j int) bool

func (Directives) Swap 

func (d Directives) Swap(i, j int)

type Field 

type Field struct {
	Name    FieldName    `json:"name"`
	Type    BaseType     `json:"type"`
	Options FieldOptions `json:"options"`

	CreatedAt int64 `json:"createdAt,omitempty"`
}

Field represents a field and its configuration.

func (*Field) CreateSQL 

func (f *Field) CreateSQL() string

CreateSQL returns the SQL representation of the field to be used in a CREATE TABLE statement.

func (*Field) IsPrimaryKey 

func (f *Field) IsPrimaryKey() bool

IsPrimaryKey returns true if the field is the primary key field (of either type ID or STRING).

func (*Field) String 

func (f *Field) String() string

String returns the field name as a string.

func (*Field) StringKeys 

func (f *Field) StringKeys() bool

StringKeys returns true if the field uses string keys.

type FieldName 

type FieldName string

FieldName is a typed string used for field names.

type FieldOptions 

type FieldOptions struct {
	Min            pql.Decimal   `json:"min,omitempty"`
	Max            pql.Decimal   `json:"max,omitempty"`
	Scale          int64         `json:"scale,omitempty"`
	NoStandardView bool          `json:"no-standard-view,omitempty"` // TODO: we should remove this
	CacheType      string        `json:"cache-type,omitempty"`
	CacheSize      uint32        `json:"cache-size,omitempty"`
	TimeUnit       string        `json:"time-unit,omitempty"`
	Epoch          time.Time     `json:"epoch,omitempty"`
	TimeQuantum    TimeQuantum   `json:"time-quantum,omitempty"`
	TTL            time.Duration `json:"ttl,omitempty"`
	ForeignIndex   string        `json:"foreign-index,omitempty"`
}

FieldOptions represents options to set when initializing a field.

type Job 

type Job string

Job is a generic identifier used to represent a specific role assigned to a worker.

type Jobs 

type Jobs []Job

Jobs is a slice of Job.

type MDSService 

type MDSService interface {
	Service
}

type MultiService 

type MultiService interface {
	Service

	SetKey(ServiceKey)
	Key() ServiceKey
}

MultiService is a service type which can have multiple instances within ServicesManager.

type Node 

type Node struct {
	Address Address `json:"address"`

	RoleTypes []RoleType `json:"role-types"`
}

Node is used in API requests, like RegisterNode (before being assigned roles).

type NodeService 

type NodeService interface {
	CreateNode(context.Context, Address, *Node) error
	ReadNode(context.Context, Address) (*Node, error)
	DeleteNode(context.Context, Address) error
	Nodes(context.Context) ([]*Node, error)
}

NodeService represents a service for managing Nodes.

type NopAddressManager 

type NopAddressManager struct{}

NopAddressManager is a no-op implementation of the AddressManager interface.

func NewNopAddressManager 

func NewNopAddressManager() *NopAddressManager

func (*NopAddressManager) AddAddresses 

func (a *NopAddressManager) AddAddresses(ctx context.Context, addrs ...Address) error

func (*NopAddressManager) RemoveAddresses 

func (a *NopAddressManager) RemoveAddresses(ctx context.Context, addrs ...Address) error

type OrganizationID 

type OrganizationID string

OrganizationID is the unique organization identifier, currently generated by the Control Plane in a FeatureBase cloud implementation. In that implementation, its value is a uuid as a string, but there's nothing enforcing that; the value could be any string.

type PartitionNum 

type PartitionNum int

PartitionNum is the numerical (int) partition value.

func (PartitionNum) String 

func (p PartitionNum) String() string

String returns the PartitionNum as a string.

type PartitionNums 

type PartitionNums []PartitionNum

PartitionNums is a slice of PartitionNum.

func (PartitionNums) Len 

func (p PartitionNums) Len() int

func (PartitionNums) Less 

func (p PartitionNums) Less(i, j int) bool

func (PartitionNums) Swap 

func (p PartitionNums) Swap(i, j int)

type QualifiedTable 

type QualifiedTable struct {
	Table
	TableQualifier
}

QualifiedTable wraps Table and includes a TableQualifier.

func NewQualifiedTable 

func NewQualifiedTable(qual TableQualifier, tbl *Table) *QualifiedTable

NewQualifiedTable returns the tbl as a QualifiedTable with the provided TableQualifier.

func (QualifiedTable) Key 

func (qt QualifiedTable) Key() TableKey

Key returns the string-encoded (delimited by TableKeyDelimiter) globally unique TableKey.

func (*QualifiedTable) QualifiedID 

func (qt *QualifiedTable) QualifiedID() QualifiedTableID

QualifiedID returns the QualifiedTableID for the table.

func (*QualifiedTable) Qualifier 

func (qt *QualifiedTable) Qualifier() TableQualifier

Qualifier returns the TableQualifer portion of the QualifiedTable.

func (QualifiedTable) String 

func (qt QualifiedTable) String() string

String returns a human-friendly version of the QualifiedTable. It is only used for display purposes; it is not used as any kind of key.

type QualifiedTableID 

type QualifiedTableID struct {
	TableQualifier
	ID   TableID   `json:"id"`
	Name TableName `json:"name"`
}

QualifiedTableID is a globally unique table identifier. It is a sub-set of a QualifiedTable (i.e. it's just the identification portion). Most things will take a Name or an ID and do the right thing™.

func NewQualifiedTableID 

func NewQualifiedTableID(q TableQualifier, id TableID) QualifiedTableID

NewQualifiedTableID is a helper function used to create a QualifiedTableID from the provided arguments.

func QualifiedTableIDFromKey 

func QualifiedTableIDFromKey(key string) (QualifiedTableID, error)

QualifiedTableIDFromKey decodes a string key into a QualifiedTableID. The key is assumed to have been encoded using the QualifiedTableID.Key() method.

func (QualifiedTableID) Equals 

func (qtid QualifiedTableID) Equals(other QualifiedTableID) bool

Equals returns true if `other` is the same as qtid. Note: the `Name` value is ignored in this comparison; only `TableQaulifer` and `ID` are considered.

func (QualifiedTableID) Key 

func (qtid QualifiedTableID) Key() TableKey

Key returns the string-encoded (delimited by TableKeyDelimiter) globally unique TableKey. The key has a prefix because FeatureBase index name restrictions require the name to start with a non-numeric value, and since a uuid can contain a number as its first character, we have to prefix it with something.

func (QualifiedTableID) String 

func (qtid QualifiedTableID) String() string

String returns a human-friendly version of the TableQualifierID. It is only used for display purposes; it is not used as any kind of key. For that, see the TableQualifierID.Key() method.

type QualifiedTables 

type QualifiedTables []*QualifiedTable

QualifiedTables is a sortable slice of QualifiedTable.

func (QualifiedTables) Len 

func (o QualifiedTables) Len() int

func (QualifiedTables) Less 

func (o QualifiedTables) Less(i, j int) bool

func (QualifiedTables) Swap 

func (o QualifiedTables) Swap(i, j int)

type QueryerService 

type QueryerService interface {
	Service

	SetMDS(Address) error
}

type Role 

type Role interface {
	Type() RoleType
}

Role is an interface for any role which a worker node can assume.

type RoleType 

type RoleType string

RoleType represents a role type which a worker node can act as. 

const (
	RoleTypeCompute   RoleType = "compute"
	RoleTypeTranslate RoleType = "translate"
)

type RoleTypes 

type RoleTypes []RoleType

RoleTypes is a list of RoleType, used primarily to introduce helper methods on the list.

func (RoleTypes) Contains 

func (rt RoleTypes) Contains(t RoleType) bool

Contains returns true if the given RoleType is in RoleTypes.

type Schema 

type Schema struct {
	Tables []*Table
}

Schema contains a list of Tables.

func (*Schema) Table 

func (s *Schema) Table(name TableName) (*Table, bool)

Table returns the table with the provided name. If a table with that name does not exist, the returned boolean will be false.

type Service 

type Service interface {
	Start() error
	Stop() error
	Address() Address
	HTTPHandler() http.Handler
}

Service is an interface implemented by any service which is part of ServiceManager.

type ServiceKey 

type ServiceKey string

ServiceKey is a unique key used to identify one service managed by the ServiceManager. These typically align with the ServicePrefix* values.

type ServiceManager 

type ServiceManager struct {

	// MDS
	MDS MDSService

	// Queryer
	Queryer QueryerService

	Logger logger.Logger
	// contains filtered or unexported fields
}

ServiceManager manages the various services running in process. It is used to do things like start/stop services, and it dynamically builds the http router depending on the state of all services.

func NewServiceManager 

func NewServiceManager() *ServiceManager

NewServiceManager returns a new ServiceManager with default values.

func (*ServiceManager) AddComputer 

func (s *ServiceManager) AddComputer(cs ComputerService) ServiceKey

AddComputer adds the provided ComputerService to ServiceManager. It assigns the service a unique ServiceKey.

func (*ServiceManager) Computer 

func (s *ServiceManager) Computer(key ServiceKey) ComputerService

Computer returns the ComputerService specified by the provided key.

func (*ServiceManager) ComputerStart 

func (s *ServiceManager) ComputerStart(key ServiceKey) error

ComputerStart starts the Computer service specified by the provided key.

func (*ServiceManager) ComputerStop 

func (s *ServiceManager) ComputerStop(key ServiceKey) error

ComputerStop stops the Computer service specified by the provided key.

func (*ServiceManager) Computers 

func (s *ServiceManager) Computers() map[ServiceKey]ComputerService

Computers returns a map (keyed by ServiceKey) of all computers registered with ServiceManager.

func (*ServiceManager) HTTPHandler 

func (s *ServiceManager) HTTPHandler() http.Handler

HTTPHandler returns the current http.Handler for ServiceManager based on the state of its services.

func (*ServiceManager) MDSStart 

func (s *ServiceManager) MDSStart() error

MDSStart starts the MDS service.

func (*ServiceManager) MDSStop 

func (s *ServiceManager) MDSStop() error

MDSStop stops the MDS service.

func (*ServiceManager) QueryerStart 

func (s *ServiceManager) QueryerStart() error

QueryerStart starts the Queryer service.

func (*ServiceManager) QueryerStop 

func (s *ServiceManager) QueryerStop() error

QueryerStop stops the Queryer service.

func (*ServiceManager) RemoveComputer 

func (s *ServiceManager) RemoveComputer(key ServiceKey) bool

RemoveComputer removes the ComputerService specified by the provided key.

func (*ServiceManager) StartAll 

func (s *ServiceManager) StartAll() error

StartAll starts all services which have been added to ServiceManager.

type Set 

type Set[K constraints.Ordered] map[K]struct{}

Set is a set of orderable items.

func NewSet 

func NewSet[K constraints.Ordered](stuff ...K) Set[K]

func (Set[K]) Add 

func (s Set[K]) Add(k K)

Add adds k to the set.

func (Set[K]) Contains 

func (s Set[K]) Contains(k K) bool

Contains returns true if k is in the set.

func (Set[K]) Copy 

func (s Set[K]) Copy() Set[K]

Copy creates a copy of the set.

func (Set[K]) Count 

func (s Set[K]) Count() int

Count returns the number of items in the set.

func (Set[K]) Merge 

func (s Set[K]) Merge(s2 Set[K])

Merge adds the members of s2 to s.

func (Set[K]) Minus 

func (s Set[K]) Minus(s2 Set[K]) Set[K]

Minus returns the a copy of s without any members which are also in s2.

func (Set[K]) Plus 

func (s Set[K]) Plus(s2 Set[K]) Set[K]

Plus returns a copy of s that also contains all members of s2.

func (Set[K]) Remove 

func (s Set[K]) Remove(k K)

Remove removes k from the set.

func (Set[K]) Slice 

func (s Set[K]) Slice() []K

Slice returns a slice containing each member of the set in an undefined order.

func (Set[K]) Sorted 

func (s Set[K]) Sorted() []K

Sorted returns Set[K] as a sorted slice of K.

type ShardNum 

type ShardNum uint64

ShardNum is the numerical (uint64) shard value.

func (ShardNum) String 

func (s ShardNum) String() string

type ShardNums 

type ShardNums []ShardNum

ShardNums is a slice of ShardNum.

type SnapshotFieldKeysRequest 

type SnapshotFieldKeysRequest struct {
	Address Address `json:"address"`

	TableKey    TableKey  `json:"table-key"`
	Field       FieldName `json:"field"`
	FromVersion int       `json:"from-version"`
	ToVersion   int       `json:"to-version"`

	Directive Directive `json:"directive"`
}

type SnapshotShardDataRequest 

type SnapshotShardDataRequest struct {
	Address Address `json:"address"`

	TableKey    TableKey `json:"table-key"`
	ShardNum    ShardNum `json:"shard"`
	FromVersion int      `json:"from-version"`
	ToVersion   int      `json:"to-version"`

	Directive Directive `json:"directive"`
}

type SnapshotTableKeysRequest 

type SnapshotTableKeysRequest struct {
	Address Address `json:"address"`

	TableKey     TableKey     `json:"table-key"`
	PartitionNum PartitionNum `json:"partition"`
	FromVersion  int          `json:"from-version"`
	ToVersion    int          `json:"to-version"`

	Directive Directive `json:"directive"`
}

type Table 

type Table struct {
	ID         TableID   `json:"id,omitempty"`
	Name       TableName `json:"name,omitempty"`
	Fields     []*Field  `json:"fields"`
	PartitionN int       `json:"partitionN"`

	Description string `json:"description,omitempty"`
	CreatedAt   int64  `json:"createdAt,omitempty"`
}

Table represents a table and its configuration.

func NewTable 

func NewTable(name TableName) *Table

NewTable returns a new instance of table with a pseudo-random ID which is assumed to be unique within the scope of a TableQualifer.

func (*Table) CreateID 

func (t *Table) CreateID() (TableID, error)

CreateID generates a unique identifier for Table. If Table has already been assigned an ID, then an error is returned.

func (*Table) CreateSQL 

func (t *Table) CreateSQL() string

CreateSQL returns the SQL CREATE TABLE string necessary to create the table.

func (*Table) Field 

func (t *Table) Field(name FieldName) (*Field, bool)

Field returns the field with the provided name. If a field with that name does not exist, the returned boolean will be false.

func (*Table) FieldNames 

func (t *Table) FieldNames() []FieldName

FieldNames returns the list of field names associated with the table.

func (*Table) HasValidPrimaryKey 

func (t *Table) HasValidPrimaryKey() bool

HasValidPrimaryKey returns false if the table does not contain a primary key field (which is required), or if the primary key field is not a valid type.

func (*Table) RemoveField 

func (t *Table) RemoveField(name FieldName) bool

RemoveField removes the given field by name. It returns true if the field was removed.

func (*Table) StringKeys 

func (t *Table) StringKeys() bool

StringKeys returns true if the table's primary key is either a string or a concatenation of fields.

type TableID 

type TableID string

TableID is a table identifier. It is unique within the scope of a TableQualifier. Coupled with a TableQualifier, it makes up a QualifiedTableID and, when encoded as a string, a TableKey.

type TableIDs 

type TableIDs []TableID

TableIDs is a sortable slice of TableID.

func (TableIDs) Len 

func (s TableIDs) Len() int

func (TableIDs) Less 

func (s TableIDs) Less(i, j int) bool

func (TableIDs) Swap 

func (s TableIDs) Swap(i, j int)

type TableKey 

type TableKey string

TableKey is a globally unique identifier for a table; it is effectively the compound key: (org, database, table). This is (hopefully) the value that will be used when interfacing with services which are unaware of table qualifiers. For example, the FeatureBase server has no notion of organization or database; its top level type is index/indexName/table. So in this case, until and unless we introduce table qualifiers into FeatureBase, we will use TableKey as the value for index.Name.

func (TableKey) QualifiedTableID 

func (tk TableKey) QualifiedTableID() QualifiedTableID

QualifiedTableID returns the QualifiedTableID based on the key. If TableKey can't be parsed into a valid (i.e. complete) QualifiedTableID, then blank values are used where necessary.

type TableKeys 

type TableKeys []TableKey

TableKeys is a sortable slice of TableKey.

func (TableKeys) Len 

func (s TableKeys) Len() int

func (TableKeys) Less 

func (s TableKeys) Less(i, j int) bool

func (TableKeys) Swap 

func (s TableKeys) Swap(i, j int)

type TableName 

type TableName string

TableName is a human-friendly string. While it is not used as a primary key, uniqueness is generally enforced within the scope of a TableQualifier.

type TableNames 

type TableNames []TableName

TableNames is a sortable slice of TableName.

func (TableNames) Len 

func (s TableNames) Len() int

func (TableNames) Less 

func (s TableNames) Less(i, j int) bool

func (TableNames) Swap 

func (s TableNames) Swap(i, j int)

type TableQualifier 

type TableQualifier struct {
	OrganizationID OrganizationID `json:"org-id"`
	DatabaseID     DatabaseID     `json:"db-id"`
}

TableQualifier contains all the elements required to fully qualify a table.

func NewTableQualifier 

func NewTableQualifier(orgID OrganizationID, dbID DatabaseID) TableQualifier

NewTableQualifier is a helper function used to create a TableQualifer from the provided arguments.

func (TableQualifier) Key 

func (tq TableQualifier) Key() TableQualifierKey

Key returns the string-encoded (delimited by TableKeyDelimiter) TableQualifierKey.

func (TableQualifier) String 

func (tq TableQualifier) String() string

String returns a human-friendly version of the TableQualifier. It is only used for display purposes; it is not used as any kind of key. For that, see the TableQualifier.Key() method and the TableQualifierKey type.

type TableQualifierKey 

type TableQualifierKey string

TableQualifierKey is the unique TableQualifer values encoded as a string. The current encoding is delimited as `prefix|OrganizationID|DatabaseID` (where the pipe may be some other delimiter) by the TableQualifier.Key() method.

func (TableQualifierKey) DatabaseID 

func (tqk TableQualifierKey) DatabaseID() DatabaseID

DatabaseID returns the DatabaseID value that has been encoded into the TableQualifierKey string.

func (TableQualifierKey) OrganizationID 

func (tqk TableQualifierKey) OrganizationID() OrganizationID

OrganizationID returns the OrganizationID value that has been encoded into the TableQualifierKey string.

func (TableQualifierKey) Qualifier 

func (tqk TableQualifierKey) Qualifier() TableQualifier

Qualifier returns the Qualifier based on the values encoded into the TableQualifierKey string.

type Tables 

type Tables []*Table

Tables is a sortable slice of Table.

func (Tables) Len 

func (o Tables) Len() int

func (Tables) Less 

func (o Tables) Less(i, j int) bool

func (Tables) Swap 

func (o Tables) Swap(i, j int)

type TimeQuantum 

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func (TimeQuantum) Granularity 

func (q TimeQuantum) Granularity() rune

func (TimeQuantum) HasDay 

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour 

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth 

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear 

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (TimeQuantum) IsEmpty 

func (q TimeQuantum) IsEmpty() bool

IsEmpty returns true if the quantum is empty.

func (*TimeQuantum) Set 

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String 

func (q TimeQuantum) String() string

String returns the TimeQuantum as a string type.

func (TimeQuantum) Type 

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid 

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type TranslateRole 

type TranslateRole struct {
	TableKey   TableKey            `json:"table-key"`
	Partitions VersionedPartitions `json:"partitions"`
	Fields     VersionedFields     `json:"fields"`
}

TranslateRole is a role specific to translate nodes.

func (*TranslateRole) Type 

func (cr *TranslateRole) Type() RoleType

Type returns the type for TransteRole. This is mainly to impolement the Role interface.

type VersionStore 

type VersionStore interface {
	AddTable(ctx context.Context, qtid QualifiedTableID) error
	RemoveTable(ctx context.Context, qtid QualifiedTableID) (VersionedShards, VersionedPartitions, error)

	// Shards (shardData)
	AddShards(ctx context.Context, qtid QualifiedTableID, shards ...VersionedShard) error
	Shards(ctx context.Context, qtid QualifiedTableID) (VersionedShards, bool, error)
	ShardVersion(ctx context.Context, qtid QualifiedTableID, shardNum ShardNum) (int, bool, error)
	ShardTables(ctx context.Context, qual TableQualifier) (TableIDs, error)

	// Partitions (tableKeys)
	AddPartitions(ctx context.Context, qtid QualifiedTableID, partitions ...VersionedPartition) error
	Partitions(ctx context.Context, qtid QualifiedTableID) (VersionedPartitions, bool, error)
	PartitionVersion(ctx context.Context, qtid QualifiedTableID, partitionNum PartitionNum) (int, bool, error)
	PartitionTables(ctx context.Context, qual TableQualifier) (TableIDs, error)

	// Fields (fieldKeys)
	AddFields(ctx context.Context, qtid QualifiedTableID, fields ...VersionedField) error
	Fields(ctx context.Context, qtid QualifiedTableID) (VersionedFields, bool, error)
	FieldVersion(ctx context.Context, qtid QualifiedTableID, field FieldName) (int, bool, error)
	FieldTables(ctx context.Context, qual TableQualifier) (TableIDs, error)

	Copy(ctx context.Context) (VersionStore, error)
}

VersionStore is an interface for tracking Shard, Partition, and Field[Key] versions. For example, when the contents of a shard are checkpointed, and a snapshot is generated, and the write log messages for that shard are truncated, the ShardVersion for that shard is incremented. The VersionStore is the interface through which various services read/write that version.

type VersionedField 

type VersionedField struct {
	Name    FieldName `json:"name"`
	Version int       `json:"version"`
}

VersionedField is used in a similar way to VersionedShard and VersionedPartition in that they all contain a snapshot version.

func NewVersionedField 

func NewVersionedField(name FieldName, version int) VersionedField

NewVersionedField returns a VersionedField with the provided name and version.

func (VersionedField) String 

func (f VersionedField) String() string

String returns the VersionedField (i.e. its Name and Version) as a string.

type VersionedFields 

type VersionedFields []VersionedField

VersionedFields is a sortable slice of VersionedField.

func (VersionedFields) Len 

func (f VersionedFields) Len() int

func (VersionedFields) Less 

func (f VersionedFields) Less(i, j int) bool

func (VersionedFields) Swap 

func (f VersionedFields) Swap(i, j int)

type VersionedPartition 

type VersionedPartition struct {
	Num     PartitionNum `json:"num"`
	Version int          `json:"version"`
}

VersionedPartition is a partition number along with the snapshot version which it is currently writing at.

func NewVersionedPartition 

func NewVersionedPartition(num PartitionNum, version int) VersionedPartition

NewVersionedPartition returns a VersionedPartition with the provided partition number and version.

func (VersionedPartition) String 

func (p VersionedPartition) String() string

String returns the VersionedPartition (i.e. its Num and Version) as a string.

type VersionedPartitions 

type VersionedPartitions []VersionedPartition

VersionedPartitions is a sortable slice of VersionedPartition.

func NewVersionedPartitions 

func NewVersionedPartitions(partitionNums ...PartitionNum) VersionedPartitions

NewVersionedPartitions returns the provided list of partition nums as a list of VersionedPartition with an invalid version (-1). This is to use for cases where the request should not be aware of a partition versioning.

func (VersionedPartitions) Len 

func (p VersionedPartitions) Len() int

func (VersionedPartitions) Less 

func (p VersionedPartitions) Less(i, j int) bool

func (VersionedPartitions) Nums 

func (p VersionedPartitions) Nums() []PartitionNum

Nums returns a slice of all the partition numbers in VersionedPartitions.

func (VersionedPartitions) Swap 

func (p VersionedPartitions) Swap(i, j int)

type VersionedShard 

type VersionedShard struct {
	Num     ShardNum `json:"num"`
	Version int      `json:"version"`
}

VersionedShard is a shard number along with the snapshot version which it is currently writing at.

func NewVersionedShard 

func NewVersionedShard(num ShardNum, version int) VersionedShard

NewVersionedShard returns a VersionedShard with the provided shard number and version.

func (VersionedShard) String 

func (s VersionedShard) String() string

String returns the VersionedShard (i.e. its Num and Version) as a string.

type VersionedShards 

type VersionedShards []VersionedShard

VersionedShards is a sortable slice of VersionedShard.

func NewVersionedShards 

func NewVersionedShards(shardNums ...ShardNum) VersionedShards

NewVersionedShards returns the provided list of shard nums as a list of VersionedShard with an invalid version (-1). This is to use for cases where the request should not be aware of shard versioning.

func (VersionedShards) Len 

func (s VersionedShards) Len() int

func (VersionedShards) Less 

func (s VersionedShards) Less(i, j int) bool

func (VersionedShards) Nums 

func (s VersionedShards) Nums() []ShardNum

Nums returns a slice of all the shard numbers in VersionedShards.

func (VersionedShards) Swap 

func (s VersionedShards) Swap(i, j int)

type Worker 

type Worker string

Worker is a generic identifier used to represent a service responsible for doing certain jobs. In the case of dax, this is typically the Address of a compute or translate node. Services such as the Balancer use Workers (as opposed to specifically using Address) in order to remain generic, and to keep the business logic between services slightly less coupled.

type WorkerDiff 

type WorkerDiff struct {
	WorkerID    Worker
	AddedJobs   []Job
	RemovedJobs []Job
}

WorkerDiff represents the changes made to a Worker following the latest event.

func (*WorkerDiff) Add 

func (w *WorkerDiff) Add(w2 WorkerDiff)

Add adds w2 to w. It panics of w and w2 don't have teh same worker ID. Any job that is added and then removed or removed and then added cancels out and won't be present after add is called.

type WorkerDiffs 

type WorkerDiffs []WorkerDiff

WorkerDiffs is a sortable slice of WorkerDiff.

func (WorkerDiffs) Len 

func (w WorkerDiffs) Len() int

func (WorkerDiffs) Less 

func (w WorkerDiffs) Less(i, j int) bool

func (WorkerDiffs) Swap 

func (w WorkerDiffs) Swap(i, j int)

type WorkerInfo 

type WorkerInfo struct {
	ID   Worker
	Jobs []Job
}

WorkerInfo reprents a Worker and the Jobs to which it has been assigned.

type WorkerInfos 

type WorkerInfos []WorkerInfo

WorkerInfos is a sortable slice of WorkerInfo.

func (WorkerInfos) Len 

func (w WorkerInfos) Len() int

func (WorkerInfos) Less 

func (w WorkerInfos) Less(i, j int) bool

func (WorkerInfos) Swap 

func (w WorkerInfos) Swap(i, j int)

type Workers 

type Workers []Worker

Workers is a sortable slice of Worker.

func (Workers) Len 

func (w Workers) Len() int

func (Workers) Less 

func (w Workers) Less(i, j int) bool

func (Workers) Swap 

func (w Workers) Swap(i, j int)

Source Files

View all Source files

Directories

Path Synopsis
Package boltdb contains the boltdb implementations of the DAX interfaces.
 Package boltdb contains the boltdb implementations of the DAX interfaces.
Package computer contains the compute-specific portions of the DAX architecture.
 Package computer contains the compute-specific portions of the DAX architecture.
service
Package inmem contains the in-memory implementation of the dax interfaces.
 Package inmem contains the in-memory implementation of the dax interfaces.
mds
Package mds provides the overall interface to Metadata Services.
 Package mds provides the overall interface to Metadata Services.
client
Package client is an HTTP client for MDS.
 Package client is an HTTP client for MDS.
controller
Package controller provides the core Controller struct.
 Package controller provides the core Controller struct.
controller/alpha
Package alpha contains inter-service implemenations of interfaces.
 Package alpha contains inter-service implemenations of interfaces.
controller/http
Package http provides the http implementation of the Director interface.
 Package http provides the http implementation of the Director interface.
controller/naive
Package naive contains a naive implementation of the Balancer interface.
 Package naive contains a naive implementation of the Balancer interface.
controller/naive/boltdb
Package boltdb contains the boltdb implementation of the Balancer interface.
 Package boltdb contains the boltdb implementation of the Balancer interface.
controller/partitioner
Package partitioner provides the Partitioner type, which provides helper methods for determining partitions based on string keys.
 Package partitioner provides the Partitioner type, which provides helper methods for determining partitions based on string keys.
http
poller
Package poller provides the core Poller struct.
 Package poller provides the core Poller struct.
schemar
Package schemar provides the core Schemar interface.
 Package schemar provides the core Schemar interface.
schemar/boltdb
Package boltdb contains the boltdb implementation of the Schemar interfaces.
 Package boltdb contains the boltdb implementation of the Schemar interfaces.
schemar/http
service
client
Package client is an HTTP client for MDS.
 Package client is an HTTP client for MDS.
http
service
test
Package snapshotter provides the core snapshotter structs.
 Package snapshotter provides the core snapshotter structs.
client
Package client contains an http implementation of the WriteLogger client.
 Package client contains an http implementation of the WriteLogger client.
http
Package test include external test apps, helper functions, and test data.
 Package test include external test apps, helper functions, and test data.
boltdb
Package writelogger provides the writelogger structs.
 Package writelogger provides the writelogger structs.
client
Package client contains an http implementation of the WriteLogger client.
 Package client contains an http implementation of the WriteLogger client.
http

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.