Documentation
¶
Index ¶
- Variables
- func Load(file io.Reader, layers ...FileInterface) error
- func LoadFile(fileName string, layers ...FileInterface) error
- func LossCrossEntropy(output, target *tensor.Tensor) (float32, *tensor.Tensor)
- func LossCrossEntropyWeighted(output, target *tensor.Tensor, weights []float32) (float32, *tensor.Tensor)
- func LossMSE(output, target *tensor.Tensor) (float32, *tensor.Tensor)
- func Save(file io.Writer, layers ...FileInterface) error
- func SaveFile(fileName string, layers ...FileInterface) error
- type Activation
- type ActivationFunction
- type Adam
- type AdamW
- type Context
- func Add(inputs []*tensor.Tensor, contexts []Context) (*tensor.Tensor, Context)
- func Concat(a []*tensor.Tensor, contexts []Context) (*tensor.Tensor, Context)
- func GeLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func MergeInner(a *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func NoActivationFunction(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func ReLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func ReplayGraph() Context
- func SiLU(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func Sigmoid(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- type DenseConfig
- type DenseLayer
- type DenseNetwork
- func (net *DenseNetwork) Forward(x *tensor.Tensor, ctx Context) (*tensor.Tensor, Context)
- func (net *DenseNetwork) Load(file io.Reader) error
- func (net *DenseNetwork) Save(file io.Writer) error
- func (net *DenseNetwork) Train(x *tensor.Tensor, target *tensor.Tensor, learningRate float32, ctx Context) float32
- type Embeddings
- type FFN
- type FileInterface
- type Gradients
- type LayerNorm
- type Linear
- type LinearParams
- type LossFunction
- type Module
- type OneActive
- func (act *OneActive) Exclude(indexes ...int)
- func (act *OneActive) Forward(input *tensor.Tensor, ctx Context) ([]int, Context)
- func (act *OneActive) Load(file io.Reader) error
- func (act *OneActive) Loss(input *tensor.Tensor, target []int) (float32, *tensor.Tensor)
- func (act *OneActive) Save(file io.Writer) error
- func (act *OneActive) Set(m *tensor.Tensor, index []int)
- type Optimizer
- type Residual
- type ResidualBlock
- type SGD
- type Sequential
- type Tape
Constants ¶
This section is empty.
Variables ¶
var Checks = false // will slow down
var ContextZero = Context{Tape: TapeZero}
ContextZero enables backprop with no graph collection.
var GeLUActivation = Activation{ Function: GeLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var NoActivation = Activation{ Function: NoActivationFunction, }
var ReLUActivation = Activation{ Function: ReLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var SiLUActivation = Activation{ Function: SiLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var SigmoidActivation = Activation{ Function: Sigmoid, LinearParams: &LinearParams{ InitHe: false, InitBias: 0, }, }
var TapeZero = func(grad *tensor.Tensor, grads *Gradients) *tensor.Tensor { if Checks && grad != nil { grad.CheckData() } return grad }
TapeZero ends a tape. With Checks, it rejects NaN gradients.
Functions ¶
func LoadFile ¶
func LoadFile(fileName string, layers ...FileInterface) error
func LossCrossEntropy ¶
Calculates softmax cross-entropy loss and gradient. Mainly used in classification tasks, meaning when selecting one (highest) from possible values.
func LossCrossEntropyWeighted ¶ added in v1.3.0
func LossMSE ¶
Calculates MSE (mean squared error) loss and gradient between output and target. Mainly used for regression tasks, meaning when predicting continuous numeric values.
func SaveFile ¶
func SaveFile(fileName string, layers ...FileInterface) error
Types ¶
type Activation ¶
type Activation struct {
Function ActivationFunction
LinearParams *LinearParams
}
Activation with parameters
type ActivationFunction ¶
Activation function
type Adam ¶ added in v1.4.0
type Adam struct {
LR float32
Beta1 float32
Beta2 float32
Eps float32
// contains filtered or unexported fields
}
Adam is bias-corrected Adam. Moments are stored on the parameter pointer and kept across Step calls.
type AdamW ¶ added in v1.4.0
AdamW is Adam with decoupled weight decay. The decay term is lr * WeightDecay * param and is not added to the gradient.
type Context ¶ added in v1.4.0
type Context struct {
Tape Tape
Graph *graph.Collector
Val graph.ValueID
OpLabel string // optional label for the next recorded op
OpPath string // optional layout path for the next recorded op
}
Context carries execution machinery for one forward pass. A tensor.Tensor carries the computation data (activations, weights, gradients). Context carries the optional autograd Tape, graph collector, and op metadata. Passed by value. WithTape, WithVal, WithLabel, and WithPath return copies.
func Concat ¶
Concats inner dimensions of tensors (for now assumes 2D) [B, C] + [B, D] + ... -> [B, C+D+...]
func MergeInner ¶
Merges two inner most dimensions of tensor
func NoActivationFunction ¶
No activation — passes through; traced as logits (raw class scores).
func ReLU ¶
ReLU activation function - Simple, cheap, works well. - Downside: derivative is zero for x<0 -> dead neurons
func ReplayGraph ¶ added in v1.3.0
func ReplayGraph() Context
ReplayGraph returns a context that records and emits ops on the active collector.
func SiLU ¶
SiLU (Swish) activation function - Smoothly transitions through zero (no sharp cutoff). - Keeps small gradient even for negative inputs. - Helps networks learn better in deeper or noisy models. - Used in EfficientNet and modern LLMs (e.g. Transformer FFNs).
func Sigmoid ¶
Sigmoid activation function - Smoothly squashes input to range [0,1] - useful for probabilities. - Commonly used for probabilities and gating. - Vanishing gradients, non-zero centered output, expensive.
func (Context) WithLabel ¶ added in v1.4.0
WithLabel sets the label used by the next graph op (e.g. linear).
func (Context) WithPath ¶ added in v1.4.0
WithPath sets the layout path used by the next graph op (e.g. attn, ffn).
type DenseConfig ¶
type DenseConfig struct {
InputNodes int // how many input nodes
HiddenNodes []int // how many hidden layers and nodes per layer
OutputNodes int // how many output nodes
HiddenActivation Activation // hidden layers activation
FinalActivation Activation // final layer activation
Loss LossFunction // loss function
}
Dense network configuration
type DenseLayer ¶
type DenseLayer struct {
Linear *Linear
Activation ActivationFunction
}
Dense layer
func NewDenseLayer ¶
func NewDenseLayer(inputs, outputs int, activation Activation) *DenseLayer
Creates new dense layer
type DenseNetwork ¶
type DenseNetwork struct {
Config DenseConfig
Layers []*DenseLayer
}
Dense network with multiple layers for easy use. You can also use dense layers separately in your custom network.
func NewDenseNetwork ¶
func NewDenseNetwork(cfg *DenseConfig) *DenseNetwork
Creates new dense network with random weights
type Embeddings ¶
type Embeddings struct {
Count int // how many embeddings
Size int // individual embedding size
Embeddings *tensor.Tensor // [index,size]
Excluded map[int]bool // excluded embeddings will not be trained
}
Embeddings are tensors where one is selectively (index) used in the network forward pass. Each embedding is backpropagated separately when part of the pass.
func NewEmbeddings ¶
func NewEmbeddings(count, size int, rep ...tensor.AllocReporter) *Embeddings
func (*Embeddings) Exclude ¶
func (emb *Embeddings) Exclude(indexes ...int)
type FFN ¶
type FFN struct {
*Sequential
}
type FileInterface ¶
type Gradients ¶ added in v1.4.0
type Gradients struct {
// contains filtered or unexported fields
}
Gradients sums parameter gradients during backward. Keys are the module parameter tensors themselves, not views of them.
func NewGradients ¶ added in v1.4.0
func NewGradients() *Gradients
type LayerNorm ¶
type LayerNorm struct {
Gamma *tensor.Tensor // scale [D]
Beta *tensor.Tensor // shift [D]
Eps float32
}
Layer Normalization makes each vector (D) to have mean 0 and variance 1
func NewLayerNorm ¶
type Linear ¶
Linear layer that computes y = x·W + b Bias is optional
func NewLinear ¶
func NewLinear(inputs, outputs int, hasBias bool, params *LinearParams, rep ...tensor.AllocReporter) *Linear
type LinearParams ¶
type LossFunction ¶
Loss function
type Module ¶
type Module interface {
Forward(*tensor.Tensor, Context) (*tensor.Tensor, Context)
Load(file io.Reader) error
Save(file io.Writer) error
}
Module is one layer. Forward takes a tensor of values and a Context of execution machinery.
type OneActive ¶
func NewOneActive ¶
type Optimizer ¶ added in v1.4.0
type Optimizer interface {
Step(g *Gradients)
}
Optimizer applies one parameter update from a backward pass.
type ResidualBlock ¶
type ResidualBlock struct {
*Residual
}
Generic residual block
f(x) = x + linear( activation( linear( norm(x) ) ) )
func NewResidualBlock ¶
func NewResidualBlock(outerSize, innerSize int, eps float32, activation Activation) *ResidualBlock
type Sequential ¶
type Sequential struct {
Modules []Module
}
Sequential operation
f(x) = Modules[...]( Modules[0](x) )
func NewSequential ¶
func NewSequential(modules ...Module) *Sequential