neural

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Checks = false // will slow down
View Source
var ContextZero = Context{Tape: TapeZero}

ContextZero enables backprop with no graph collection.

View Source
var GeLUActivation = Activation{
	Function: GeLU,
	LinearParams: &LinearParams{
		InitHe:   true,
		InitBias: 0.01,
	},
}
View Source
var NoActivation = Activation{
	Function: NoActivationFunction,
}
View Source
var ReLUActivation = Activation{
	Function: ReLU,
	LinearParams: &LinearParams{
		InitHe:   true,
		InitBias: 0.01,
	},
}
View Source
var SiLUActivation = Activation{
	Function: SiLU,
	LinearParams: &LinearParams{
		InitHe:   true,
		InitBias: 0.01,
	},
}
View Source
var SigmoidActivation = Activation{
	Function: Sigmoid,
	LinearParams: &LinearParams{
		InitHe:   false,
		InitBias: 0,
	},
}
View Source
var TapeZero = func(grad *tensor.Tensor, grads *Gradients) *tensor.Tensor {
	if Checks && grad != nil {
		grad.CheckData()
	}
	return grad
}

TapeZero ends a tape. With Checks, it rejects NaN gradients.

Functions

func Load

func Load(file io.Reader, layers ...FileInterface) error

func LoadFile

func LoadFile(fileName string, layers ...FileInterface) error

func LossCrossEntropy

func LossCrossEntropy(output, target *tensor.Tensor) (float32, *tensor.Tensor)

Calculates softmax cross-entropy loss and gradient. Mainly used in classification tasks, meaning when selecting one (highest) from possible values.

func LossCrossEntropyWeighted added in v1.3.0

func LossCrossEntropyWeighted(output, target *tensor.Tensor, weights []float32) (float32, *tensor.Tensor)

func LossMSE

func LossMSE(output, target *tensor.Tensor) (float32, *tensor.Tensor)

Calculates MSE (mean squared error) loss and gradient between output and target. Mainly used for regression tasks, meaning when predicting continuous numeric values.

func Save

func Save(file io.Writer, layers ...FileInterface) error

func SaveFile

func SaveFile(fileName string, layers ...FileInterface) error

Types

type Activation

type Activation struct {
	Function     ActivationFunction
	LinearParams *LinearParams
}

Activation with parameters

func (Activation) Forward

func (a Activation) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

func (Activation) Load

func (a Activation) Load(file io.Reader) error

func (Activation) Save

func (a Activation) Save(file io.Writer) error

type ActivationFunction

type ActivationFunction func(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

Activation function

type Adam added in v1.4.0

type Adam struct {
	LR    float32
	Beta1 float32
	Beta2 float32
	Eps   float32
	// contains filtered or unexported fields
}

Adam is bias-corrected Adam. Moments are stored on the parameter pointer and kept across Step calls.

func NewAdam added in v1.4.0

func NewAdam(lr float32) *Adam

func (*Adam) Step added in v1.4.0

func (o *Adam) Step(g *Gradients)

type AdamW added in v1.4.0

type AdamW struct {
	Adam
	WeightDecay float32
}

AdamW is Adam with decoupled weight decay. The decay term is lr * WeightDecay * param and is not added to the gradient.

func NewAdamW added in v1.4.0

func NewAdamW(lr float32) *AdamW

func (*AdamW) Step added in v1.4.0

func (o *AdamW) Step(g *Gradients)

type Context added in v1.4.0

type Context struct {
	Tape    Tape
	Graph   *graph.Collector
	Val     graph.ValueID
	OpLabel string // optional label for the next recorded op
	OpPath  string // optional layout path for the next recorded op
}

Context carries execution machinery for one forward pass. A tensor.Tensor carries the computation data (activations, weights, gradients). Context carries the optional autograd Tape, graph collector, and op metadata. Passed by value. WithTape, WithVal, WithLabel, and WithPath return copies.

func Add

func Add(inputs []*tensor.Tensor, contexts []Context) (*tensor.Tensor, Context)

Adds same size tensors

func Concat

func Concat(a []*tensor.Tensor, contexts []Context) (*tensor.Tensor, Context)

Concats inner dimensions of tensors (for now assumes 2D) [B, C] + [B, D] + ... -> [B, C+D+...]

func GeLU

func GeLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

GeLU activation function (tanh approximation variant)

func MergeInner

func MergeInner(a *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

Merges two inner most dimensions of tensor

func NoActivationFunction

func NoActivationFunction(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

No activation — passes through; traced as logits (raw class scores).

func ReLU

func ReLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

ReLU activation function - Simple, cheap, works well. - Downside: derivative is zero for x<0 -> dead neurons

func ReplayGraph added in v1.3.0

func ReplayGraph() Context

ReplayGraph returns a context that records and emits ops on the active collector.

func SiLU

func SiLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

SiLU (Swish) activation function - Smoothly transitions through zero (no sharp cutoff). - Keeps small gradient even for negative inputs. - Helps networks learn better in deeper or noisy models. - Used in EfficientNet and modern LLMs (e.g. Transformer FFNs).

func Sigmoid

func Sigmoid(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

Sigmoid activation function - Smoothly squashes input to range [0,1] - useful for probabilities. - Commonly used for probabilities and gating. - Vanishing gradients, non-zero centered output, expensive.

func (Context) WithLabel added in v1.4.0

func (ctx Context) WithLabel(label string) Context

WithLabel sets the label used by the next graph op (e.g. linear).

func (Context) WithPath added in v1.4.0

func (ctx Context) WithPath(path string) Context

WithPath sets the layout path used by the next graph op (e.g. attn, ffn).

func (Context) WithTape added in v1.4.0

func (ctx Context) WithTape(tape Tape) Context

WithTape returns a copy with an updated tape chain.

func (Context) WithVal added in v1.4.0

func (ctx Context) WithVal(v graph.ValueID) Context

WithVal returns a copy pointing at a different graph value id.

type DenseConfig

type DenseConfig struct {
	InputNodes       int          // how many input nodes
	HiddenNodes      []int        // how many hidden layers and nodes per layer
	OutputNodes      int          // how many output nodes
	HiddenActivation Activation   // hidden layers activation
	FinalActivation  Activation   // final layer activation
	Loss             LossFunction // loss function
}

Dense network configuration

type DenseLayer

type DenseLayer struct {
	Linear     *Linear
	Activation ActivationFunction
}

Dense layer

func NewDenseLayer

func NewDenseLayer(inputs, outputs int, activation Activation) *DenseLayer

Creates new dense layer

func (*DenseLayer) Forward

func (layer *DenseLayer) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

Forwards layer

func (*DenseLayer) Load

func (layer *DenseLayer) Load(file io.Reader) error

func (*DenseLayer) Save

func (layer *DenseLayer) Save(file io.Writer) error

type DenseNetwork

type DenseNetwork struct {
	Config DenseConfig
	Layers []*DenseLayer
}

Dense network with multiple layers for easy use. You can also use dense layers separately in your custom network.

func NewDenseNetwork

func NewDenseNetwork(cfg *DenseConfig) *DenseNetwork

Creates new dense network with random weights

func (*DenseNetwork) Forward

func (net *DenseNetwork) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

Forwards network using specified input. Input/output shapes: [batches, length]

func (*DenseNetwork) Load

func (net *DenseNetwork) Load(file io.Reader) error

func (*DenseNetwork) Save

func (net *DenseNetwork) Save(file io.Writer) error

func (*DenseNetwork) Train

func (net *DenseNetwork) Train(x *tensor.Tensor, target *tensor.Tensor, learningRate float32, ctx Context) float32

Trains one step. Backward records gradients, then SGD is applied. Returns loss. Input/output shapes: [batches, length]

type Embeddings

type Embeddings struct {
	Count      int            // how many embeddings
	Size       int            // individual embedding size
	Embeddings *tensor.Tensor // [index,size]
	Excluded   map[int]bool   // excluded embeddings will not be trained
}

Embeddings are tensors where one is selectively (index) used in the network forward pass. Each embedding is backpropagated separately when part of the pass.

func NewEmbeddings

func NewEmbeddings(count, size int, rep ...tensor.AllocReporter) *Embeddings

func (*Embeddings) Exclude

func (emb *Embeddings) Exclude(indexes ...int)

func (*Embeddings) Forward

func (emb *Embeddings) Forward(x [][]int, ctx Context) (*tensor.Tensor, Context)

Forward: index [B,T] -> embeddings [B,T,D]

func (*Embeddings) Load

func (emb *Embeddings) Load(file io.Reader) error

func (*Embeddings) Save

func (emb *Embeddings) Save(file io.Writer) error

type FFN

type FFN struct {
	*Sequential
}

func NewFFN

func NewFFN(embeddingSize, hiddenNodes int) *FFN

type FileInterface

type FileInterface interface {
	Load(file io.Reader) error
	Save(file io.Writer) error
}

type Gradients added in v1.4.0

type Gradients struct {
	// contains filtered or unexported fields
}

Gradients sums parameter gradients during backward. Keys are the module parameter tensors themselves, not views of them.

func NewGradients added in v1.4.0

func NewGradients() *Gradients

func (*Gradients) Add added in v1.4.0

func (g *Gradients) Add(param, grad *tensor.Tensor)

Add accumulates grad into param. The stored tensor is an owned copy, so it stays valid after the caller returns.

func (*Gradients) ApplySGD added in v1.4.0

func (g *Gradients) ApplySGD(learningRate float32)

ApplySGD applies param -= learningRate * grad. Gradients are kept, so a second call steps again.

type LayerNorm

type LayerNorm struct {
	Gamma *tensor.Tensor // scale [D]
	Beta  *tensor.Tensor // shift [D]
	Eps   float32
}

Layer Normalization makes each vector (D) to have mean 0 and variance 1

func NewLayerNorm

func NewLayerNorm(d int, eps float32) *LayerNorm

func (*LayerNorm) Forward

func (ln *LayerNorm) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

expects [B,T,D] or [B,D]

func (*LayerNorm) Load

func (ln *LayerNorm) Load(file io.Reader) error

func (*LayerNorm) Save

func (ln *LayerNorm) Save(file io.Writer) error

type Linear

type Linear struct {
	Weights *tensor.Tensor // weights
	Bias    *tensor.Tensor // bias (optional)
}

Linear layer that computes y = x·W + b Bias is optional

func NewLinear

func NewLinear(inputs, outputs int, hasBias bool, params *LinearParams, rep ...tensor.AllocReporter) *Linear

func (*Linear) Forward

func (l *Linear) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

func (*Linear) Load

func (l *Linear) Load(file io.Reader) error

func (*Linear) Save

func (l *Linear) Save(file io.Writer) error

type LinearParams

type LinearParams struct {
	InitStd  float32 // Initialize normal
	InitHe   bool    // Initialize with He (Kaiming) uniform distribution
	InitBias float32 // Inititialize bias value
}

type LossFunction

type LossFunction func(output, target *tensor.Tensor) (float32, *tensor.Tensor)

Loss function

type Module

type Module interface {
	Forward(*tensor.Tensor, Context) (*tensor.Tensor, Context)
	Load(file io.Reader) error
	Save(file io.Writer) error
}

Module is one layer. Forward takes a tensor of values and a Context of execution machinery.

type OneActive

type OneActive struct {
	Length   int
	Excluded map[int]bool // excluded index will not be selected
}

func NewOneActive

func NewOneActive(length int) *OneActive

func (*OneActive) Exclude

func (act *OneActive) Exclude(indexes ...int)

func (*OneActive) Forward

func (act *OneActive) Forward(input *tensor.Tensor, ctx Context) ([]int, Context)

func (*OneActive) Load

func (act *OneActive) Load(file io.Reader) error

func (*OneActive) Loss

func (act *OneActive) Loss(input *tensor.Tensor, target []int) (float32, *tensor.Tensor)

Calculates loss and gradient between input and target

func (*OneActive) Save

func (act *OneActive) Save(file io.Writer) error

func (*OneActive) Set

func (act *OneActive) Set(m *tensor.Tensor, index []int)

type Optimizer added in v1.4.0

type Optimizer interface {
	Step(g *Gradients)
}

Optimizer applies one parameter update from a backward pass.

func NewOptimizer added in v1.4.0

func NewOptimizer(name string, lr float32) (Optimizer, error)

NewOptimizer builds sgd, adam, or adamw. An empty name is SGD.

type Residual

type Residual struct {
	Norm   *LayerNorm
	Module Module
}

Residual wrapper

f(x) = x + module( norm(x) )

func NewResidual

func NewResidual(mod Module, d int, eps float32) *Residual

func (*Residual) Forward

func (r *Residual) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

func (*Residual) Load

func (r *Residual) Load(file io.Reader) error

func (*Residual) Save

func (r *Residual) Save(file io.Writer) error

type ResidualBlock

type ResidualBlock struct {
	*Residual
}

Generic residual block

f(x) = x + linear( activation( linear( norm(x) ) ) )

func NewResidualBlock

func NewResidualBlock(outerSize, innerSize int, eps float32, activation Activation) *ResidualBlock

type SGD added in v1.4.0

type SGD struct {
	LR float32
}

SGD is param -= LR * grad.

func NewSGD added in v1.4.0

func NewSGD(lr float32) *SGD

func (*SGD) Step added in v1.4.0

func (o *SGD) Step(g *Gradients)

type Sequential

type Sequential struct {
	Modules []Module
}

Sequential operation

f(x) = Modules[...]( Modules[0](x) )

func NewSequential

func NewSequential(modules ...Module) *Sequential

func (*Sequential) Forward

func (s *Sequential) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)

func (*Sequential) Load

func (s *Sequential) Load(file io.Reader) error

func (*Sequential) Save

func (s *Sequential) Save(file io.Writer) error

type Tape

type Tape func(grad *tensor.Tensor, grads *Gradients) *tensor.Tensor

Tape is one closure in the reverse-mode chain built during forward. grad is the upstream gradient. grads collects parameter gradients and is shared by the whole backward pass. The closure returns dL/d(input).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL