normalization

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2025 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package normalization provides various normalization layers for neural networks.

Package normalization provides various normalization layers for neural networks.

Package normalization provides normalization layers for the Zerfoo model.

Package normalization provides normalization layers for the Zerfoo model.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildRMSNorm added in v0.3.0 

func BuildRMSNorm[T tensor.Numeric](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	name string,
	params map[string]*graph.Parameter[T],
	attributes map[string]interface{},
) (graph.Node[T], error)

BuildRMSNorm constructs an RMSNorm node from the provided parameter and epsilon attribute.

func BuildSimplifiedLayerNormalization added in v0.3.0 

func BuildSimplifiedLayerNormalization[T tensor.Numeric](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	name string,
	params map[string]*graph.Parameter[T],
	attributes map[string]interface{},
) (graph.Node[T], error)

BuildSimplifiedLayerNormalization constructs a SimplifiedLayerNormalization node, attempting multiple common naming patterns to resolve the gain/weight parameter.

func BuildSkipSimplifiedLayerNormalization added in v0.3.0 

func BuildSkipSimplifiedLayerNormalization[T tensor.Numeric](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	name string,
	params map[string]*graph.Parameter[T],
	attributes map[string]interface{},
) (graph.Node[T], error)

BuildSkipSimplifiedLayerNormalization constructs a SkipSimplifiedLayerNormalization node, resolving the gain/weight parameter using several naming conventions.

Types

type LayerNormalization 

type LayerNormalization[T tensor.Numeric] struct {
	// contains filtered or unexported fields
}

LayerNormalization implements the Layer Normalization operation.

func NewLayerNormalization 

func NewLayerNormalization[T tensor.Numeric](engine compute.Engine[T], featureDim int, options ...LayerNormalizationOption[T]) (*LayerNormalization[T], error)

NewLayerNormalization creates a new LayerNormalization layer. featureDim: The dimension over which to normalize (typically the last dimension).

func (*LayerNormalization[T]) Attributes added in v0.3.0 

func (ln *LayerNormalization[T]) Attributes() map[string]interface{}

Attributes returns the attributes of the LayerNormalization layer.

func (*LayerNormalization[T]) Backward 

func (ln *LayerNormalization[T]) Backward(ctx context.Context, mode types.BackwardMode, dOut *tensor.TensorNumeric[T], inputs ...*tensor.TensorNumeric[T]) ([]*tensor.TensorNumeric[T], error)

Backward computes the gradients for LayerNormalization.

func (*LayerNormalization[T]) Forward 

func (ln *LayerNormalization[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward computes the Layer Normalization.

func (*LayerNormalization[T]) OpType added in v0.3.0 

func (ln *LayerNormalization[T]) OpType() string

OpType returns the operation type of the LayerNormalization layer.

func (*LayerNormalization[T]) OutputShape 

func (ln *LayerNormalization[T]) OutputShape() []int

OutputShape returns the output shape, which is the same as the input shape.

func (*LayerNormalization[T]) Parameters 

func (ln *LayerNormalization[T]) Parameters() []*graph.Parameter[T]

Parameters returns the trainable gamma and beta parameters.

type LayerNormalizationOption added in v0.3.0 

type LayerNormalizationOption[T tensor.Numeric] func(*LayerNormalizationOptions[T])

LayerNormalizationOption is a functional option for configuring LayerNormalization layers.

func WithLayerNormEpsilon added in v0.3.0 

func WithLayerNormEpsilon[T tensor.Numeric](epsilon T) LayerNormalizationOption[T]

WithLayerNormEpsilon sets the epsilon parameter for LayerNormalization.

type LayerNormalizationOptions added in v0.3.0 

type LayerNormalizationOptions[T tensor.Numeric] struct {
	Epsilon T // Small constant to avoid division by zero
}

LayerNormalizationOptions holds configuration options for LayerNormalization layers.

type RMSNorm added in v0.3.0 

type RMSNorm[T tensor.Numeric] struct {
	// contains filtered or unexported fields
}

RMSNorm is a struct that implements the graph.Node interface for RMSNorm.

func NewRMSNorm added in v0.3.0 

func NewRMSNorm[T tensor.Numeric](name string, engine compute.Engine[T], ops numeric.Arithmetic[T], modelDim int, options ...RMSNormOption[T]) (*RMSNorm[T], error)

NewRMSNorm creates a new RMSNorm layer.

func NewRMSNormFromParam added in v0.3.0 

func NewRMSNormFromParam[T tensor.Numeric](engine compute.Engine[T], ops numeric.Arithmetic[T], epsilon T, gain *graph.Parameter[T]) (*RMSNorm[T], error)

NewRMSNormFromParam creates a new RMSNorm layer from an existing gain parameter.

func (*RMSNorm[T]) Attributes added in v0.3.0 

func (r *RMSNorm[T]) Attributes() map[string]interface{}

Attributes returns the attributes.

func (*RMSNorm[T]) Backward added in v0.3.0 

func (r *RMSNorm[T]) Backward(ctx context.Context, mode types.BackwardMode, dOut *tensor.TensorNumeric[T], inputs ...*tensor.TensorNumeric[T]) ([]*tensor.TensorNumeric[T], error)

Backward computes the backward pass of the RMSNorm layer.

func (*RMSNorm[T]) Forward added in v0.3.0 

func (r *RMSNorm[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward computes the forward pass of the RMSNorm layer.

func (*RMSNorm[T]) OpType added in v0.3.0 

func (r *RMSNorm[T]) OpType() string

OpType returns the operation type.

func (*RMSNorm[T]) OutputShape added in v0.3.0 

func (r *RMSNorm[T]) OutputShape() []int

OutputShape returns the output shape of the RMSNorm layer.

func (*RMSNorm[T]) Parameters added in v0.3.0 

func (r *RMSNorm[T]) Parameters() []*graph.Parameter[T]

Parameters returns the learnable parameters of the RMSNorm layer.

func (*RMSNorm[T]) SetName added in v0.3.0 

func (r *RMSNorm[T]) SetName(name string)

SetName sets the name of the RMSNorm layer.

type RMSNormOption added in v0.3.0 

type RMSNormOption[T tensor.Numeric] func(*RMSNormOptions[T])

RMSNormOption is a functional option for configuring RMSNorm layers.

func WithRMSNormEpsilon added in v0.3.0 

func WithRMSNormEpsilon[T tensor.Numeric](epsilon T) RMSNormOption[T]

WithRMSNormEpsilon sets the epsilon parameter for RMSNorm.

type RMSNormOptions added in v0.3.0 

type RMSNormOptions[T tensor.Numeric] struct {
	Epsilon T // Small constant to avoid division by zero
}

RMSNormOptions holds configuration options for RMSNorm layers.

type SimplifiedLayerNormalization added in v0.3.0 

type SimplifiedLayerNormalization[T tensor.Numeric] struct {
	// contains filtered or unexported fields
}

SimplifiedLayerNormalization implements a simplified version of layer normalization.

func NewSimplifiedLayerNormalization added in v0.3.0 

func NewSimplifiedLayerNormalization[T tensor.Numeric](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	gain *tensor.TensorNumeric[T],
	epsilon T,
) (*SimplifiedLayerNormalization[T], error)

NewSimplifiedLayerNormalization creates a new SimplifiedLayerNormalization layer.

func (*SimplifiedLayerNormalization[T]) Attributes added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) Attributes() map[string]interface{}

Attributes returns the attributes of the SimplifiedLayerNormalization layer.

func (*SimplifiedLayerNormalization[T]) Backward added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) Backward(ctx context.Context, mode types.BackwardMode, outputGradient *tensor.TensorNumeric[T], inputs ...*tensor.TensorNumeric[T]) ([]*tensor.TensorNumeric[T], error)

Backward applies the backward pass of the SimplifiedLayerNormalization layer.

func (*SimplifiedLayerNormalization[T]) Forward added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward applies the forward pass of the SimplifiedLayerNormalization layer.

func (*SimplifiedLayerNormalization[T]) OpType added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) OpType() string

OpType returns the operation type of the SimplifiedLayerNormalization layer.

func (*SimplifiedLayerNormalization[T]) OutputShape added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) OutputShape() []int

OutputShape returns the output shape of the layer.

func (*SimplifiedLayerNormalization[T]) Parameters added in v0.3.0 

func (sln *SimplifiedLayerNormalization[T]) Parameters() []*graph.Parameter[T]

Parameters returns the learnable parameters of the layer.

type SkipSimplifiedLayerNormalization added in v0.3.0 

type SkipSimplifiedLayerNormalization[T tensor.Numeric] struct {
	// contains filtered or unexported fields
}

SkipSimplifiedLayerNormalization applies SimplifiedLayerNormalization and adds a residual connection.

func NewSkipSimplifiedLayerNormalization added in v0.3.0 

func NewSkipSimplifiedLayerNormalization[T tensor.Numeric](
	engine compute.Engine[T],
	ops numeric.Arithmetic[T],
	gain *tensor.TensorNumeric[T],
	epsilon T,
) (*SkipSimplifiedLayerNormalization[T], error)

NewSkipSimplifiedLayerNormalization creates a new SkipSimplifiedLayerNormalization layer.

func (*SkipSimplifiedLayerNormalization[T]) Attributes added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) Attributes() map[string]interface{}

Attributes returns the attributes of the underlying normalization layer.

func (*SkipSimplifiedLayerNormalization[T]) Backward added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) Backward(ctx context.Context, mode types.BackwardMode, outputGrad *tensor.TensorNumeric[T], inputs ...*tensor.TensorNumeric[T]) ([]*tensor.TensorNumeric[T], error)

Backward applies the backward pass of the SkipSimplifiedLayerNormalization layer.

func (*SkipSimplifiedLayerNormalization[T]) Forward added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) Forward(ctx context.Context, inputs ...*tensor.TensorNumeric[T]) (*tensor.TensorNumeric[T], error)

Forward applies the forward pass of the SkipSimplifiedLayerNormalization layer.

func (*SkipSimplifiedLayerNormalization[T]) OpType added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) OpType() string

OpType returns the operation type of the SkipSimplifiedLayerNormalization layer.

func (*SkipSimplifiedLayerNormalization[T]) OutputShape added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) OutputShape() []int

OutputShape returns the output shape of the layer.

func (*SkipSimplifiedLayerNormalization[T]) Parameters added in v0.3.0 

func (sln *SkipSimplifiedLayerNormalization[T]) Parameters() []*graph.Parameter[T]

Parameters returns the learnable parameters of the layer.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.