Documentation
¶
Index ¶
- Variables
- func Load(file io.Reader, layers ...FileInterface) error
- func LoadFile(fileName string, layers ...FileInterface) error
- func LossCrossEntropy(output, target *tensor.Tensor) (float32, *tensor.Tensor)
- func LossCrossEntropyWeighted(output, target *tensor.Tensor, weights []float32) (float32, *tensor.Tensor)
- func LossMSE(output, target *tensor.Tensor) (float32, *tensor.Tensor)
- func Save(file io.Writer, layers ...FileInterface) error
- func SaveFile(fileName string, layers ...FileInterface) error
- type Activation
- type ActivationFunction
- type DenseConfig
- type DenseLayer
- type DenseNetwork
- func (net *DenseNetwork) Forward(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func (net *DenseNetwork) Load(file io.Reader) error
- func (net *DenseNetwork) Save(file io.Writer) error
- func (net *DenseNetwork) Train(x *tensor.Tensor, target *tensor.Tensor, learningRate float32, tr Trace) float32
- type Embeddings
- type FFN
- type FileInterface
- type LayerNorm
- type Linear
- type LinearParams
- type LossFunction
- type Module
- type OneActive
- func (act *OneActive) Exclude(indexes ...int)
- func (act *OneActive) Forward(input *tensor.Tensor, tr Trace) ([]int, Trace)
- func (act *OneActive) Load(file io.Reader) error
- func (act *OneActive) Loss(input *tensor.Tensor, target []int) (float32, *tensor.Tensor)
- func (act *OneActive) Save(file io.Writer) error
- func (act *OneActive) Set(m *tensor.Tensor, index []int)
- type Residual
- type ResidualBlock
- type Sequential
- type Tape
- type Trace
- func Add(inputs []*tensor.Tensor, traces []Trace) (*tensor.Tensor, Trace)
- func Concat(a []*tensor.Tensor, traces []Trace) (*tensor.Tensor, Trace)
- func GeLU(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func MergeInner(a *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func NoActivationFunction(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func ReLU(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func ReplayGraph() Trace
- func SiLU(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
- func Sigmoid(x *tensor.Tensor, tr Trace) (*tensor.Tensor, Trace)
Constants ¶
This section is empty.
Variables ¶
var Checks = false // will slow down
var GeLUActivation = Activation{ Function: GeLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var NoActivation = Activation{ Function: NoActivationFunction, }
var ReLUActivation = Activation{ Function: ReLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var SiLUActivation = Activation{ Function: SiLU, LinearParams: &LinearParams{ InitHe: true, InitBias: 0.01, }, }
var SigmoidActivation = Activation{ Function: Sigmoid, LinearParams: &LinearParams{ InitHe: false, InitBias: 0, }, }
var TapeZero = func(grad *tensor.Tensor, learningRate float32) *tensor.Tensor { if Checks && grad != nil { grad.CheckData() } return grad }
Zero tape method
var TraceZero = Trace{Tape: TapeZero}
TraceZero enables backprop with no graph collection.
Functions ¶
func LoadFile ¶
func LoadFile(fileName string, layers ...FileInterface) error
func LossCrossEntropy ¶
Calculates softmax cross-entropy loss and gradient. Mainly used in classification tasks, meaning when selecting one (highest) from possible values.
func LossCrossEntropyWeighted ¶ added in v1.3.0
func LossMSE ¶
Calculates MSE (mean squared error) loss and gradient between output and target. Mainly used for regression tasks, meaning when predicting continuous numeric values.
func SaveFile ¶
func SaveFile(fileName string, layers ...FileInterface) error
Types ¶
type Activation ¶
type Activation struct {
Function ActivationFunction
LinearParams *LinearParams
}
Activation with parameters
type ActivationFunction ¶
Activation function
type DenseConfig ¶
type DenseConfig struct {
InputNodes int // how many input nodes
HiddenNodes []int // how many hidden layers and nodes per layer
OutputNodes int // how many output nodes
HiddenActivation Activation // hidden layers activation
FinalActivation Activation // final layer activation
Loss LossFunction // loss function
}
Dense network configuration
type DenseLayer ¶
type DenseLayer struct {
Linear *Linear
Activation ActivationFunction
}
Dense layer
func NewDenseLayer ¶
func NewDenseLayer(inputs, outputs int, activation Activation) *DenseLayer
Creates new dense layer
type DenseNetwork ¶
type DenseNetwork struct {
Config DenseConfig
Layers []*DenseLayer
}
Dense network with multiple layers for easy use. You can also use dense layers separately in your custom network.
func NewDenseNetwork ¶
func NewDenseNetwork(cfg *DenseConfig) *DenseNetwork
Creates new dense network with random weights
type Embeddings ¶
type Embeddings struct {
Count int // how many embeddings
Size int // individual embedding size
Embeddings *tensor.Tensor // [index,size]
Excluded map[int]bool // excluded embeddings will not be trained
}
Embeddings are tensors where one is selectively (index) used in the network forward pass. Each embedding is backpropagated separately when part of the pass.
func NewEmbeddings ¶
func NewEmbeddings(count, size int, rep ...tensor.AllocReporter) *Embeddings
func (*Embeddings) Exclude ¶
func (emb *Embeddings) Exclude(indexes ...int)
type FFN ¶
type FFN struct {
*Sequential
}
type FileInterface ¶
type LayerNorm ¶
type LayerNorm struct {
Gamma *tensor.Tensor // scale [D]
Beta *tensor.Tensor // shift [D]
Eps float32
}
Layer Normalization makes each vector (D) to have mean 0 and variance 1
func NewLayerNorm ¶
type Linear ¶
Linear layer that computes y = x·W + b Bias is optional
func NewLinear ¶
func NewLinear(inputs, outputs int, hasBias bool, params *LinearParams, rep ...tensor.AllocReporter) *Linear
type LinearParams ¶
type LossFunction ¶
Loss function
type Module ¶
type Module interface {
Forward(*tensor.Tensor, Trace) (*tensor.Tensor, Trace)
Load(file io.Reader) error
Save(file io.Writer) error
}
General module interface
type OneActive ¶
func NewOneActive ¶
type ResidualBlock ¶
type ResidualBlock struct {
*Residual
}
Generic residual block
f(x) = x + linear( activation( linear( norm(x) ) ) )
func NewResidualBlock ¶
func NewResidualBlock(outerSize, innerSize int, eps float32, activation Activation) *ResidualBlock
type Sequential ¶
type Sequential struct {
Modules []Module
}
Sequential operation
f(x) = Modules[...]( Modules[0](x) )
func NewSequential ¶
func NewSequential(modules ...Module) *Sequential
type Tape ¶
Tape records during forward pass for automatic backpropagation as chain of closures. 'grad' is gradient from next module flowing backwards
type Trace ¶ added in v1.3.0
type Trace struct {
Tape Tape
Graph *graph.Collector
Val graph.ValueID
OpLabel string // optional label for the next recorded op
OpPath string // optional layout path for the next recorded op
}
Trace carries optional autograd tape, graph collector, and current value id.
func Concat ¶
Concats inner dimensions of tensors (for now assumes 2D) [B, C] + [B, D] + ... -> [B, C+D+...]
func MergeInner ¶
Merges two inner most dimensions of tensor
func NoActivationFunction ¶
No activation — passes through; traced as logits (raw class scores).
func ReLU ¶
ReLU activation function - Simple, cheap, works well. - Downside: derivative is zero for x<0 -> dead neurons
func ReplayGraph ¶ added in v1.3.0
func ReplayGraph() Trace
ReplayGraph returns a trace that records and emits ops on the active collector.
func SiLU ¶
SiLU (Swish) activation function - Smoothly transitions through zero (no sharp cutoff). - Keeps small gradient even for negative inputs. - Helps networks learn better in deeper or noisy models. - Used in EfficientNet and modern LLMs (e.g. Transformer FFNs).
func Sigmoid ¶
Sigmoid activation function - Smoothly squashes input to range [0,1] - useful for probabilities. - Commonly used for probabilities and gating. - Vanishing gradients, non-zero centered output, expensive.
func (Trace) WithLabel ¶ added in v1.3.0
WithLabel sets the label used by the next graph op (e.g. linear).
func (Trace) WithPath ¶ added in v1.3.0
WithPath sets the layout path used by the next graph op (e.g. attn, ffn).
Source Files