learn

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source
const (
	TokenPAD = 0
	TokenUNK = 1
	TokenBOS = 2
	TokenEOS = 3
)

Variables

This section is empty.

Functions

func Learn

func Learn(u ui.UI, args []string)

Learn is the CLI entry for from-scratch GPT training.

learn [toy|path] [resume] [sgd|adam|adamw]

learn              train on The War of the Worlds with SGD
learn toy          overfit a tiny repeated corpus
learn <path>       train on a custom text file
learn resume       resume from data/checkpoints/learn
learn toy resume   resume toy run from data/checkpoints/learn-toy
learn toy adam     same run with Adam (default lr 1e-3)
learn toy adamw    AdamW, weight decay 0.01, same default lr

func NextTokenLoss

func NextTokenLoss(logits *tensor.Tensor, targets [][]int) (float32, *tensor.Tensor)

NextTokenLoss is mean cross-entropy over all positions. logits [B,T,V], targets [B][T] class indexes (next token at each position).

func Run

func Run(u ui.UI, opts TrainOpts) error

Run trains a small GPT on a text corpus.

Types

type BPEProgress

type BPEProgress func(vocabNow, vocabTarget int)

BPEProgress is called during TrainBPE with current/target vocab size.

type Block

type Block struct {
	Attn *neural.Residual
	FFN  *neural.Residual
}

Block is one pre-norm transformer block with taped residuals.

func (*Block) Forward

func (b *Block) Forward(x *tensor.Tensor, ctx neural.Context) (*tensor.Tensor, neural.Context)

func (*Block) Load

func (b *Block) Load(r io.Reader) error

func (*Block) Save

func (b *Block) Save(w io.Writer) error

type Config

type Config struct {
	VocabSize  int
	ContextLen int // T
	EmbedSize  int // D
	Layers     int
	Heads      int
	MLPSize    int
	BatchSize  int
	Eps        float32
}

Config holds hyperparameters for a small GPT-style decoder.

func DefaultConfig

func DefaultConfig(vocabSize int) Config

DefaultConfig returns small training defaults.

func ToyConfig

func ToyConfig(vocabSize int) Config

ToyConfig is small enough to overfit a tiny corpus quickly.

type EncodeProgress

type EncodeProgress func(mergesDone, mergesTotal int)

EncodeProgress reports encoding progress (merges applied / total merges).

type Model

type Model struct {
	Cfg    Config
	TokEmb *neural.Embeddings
	PosEmb *llm.PositionEmbeddings
	Blocks []*Block
	LN     *neural.LayerNorm
	LMHead *neural.Linear
}

Model is a GPT-style decoder-only transformer for next-token training.

func NewModel

func NewModel(cfg Config) *Model

func (*Model) Forward

func (m *Model) Forward(tokens [][]int, ctx neural.Context) (*tensor.Tensor, neural.Context)

Forward maps token ids [B][T] to logits [B,T,V].

func (*Model) Load

func (m *Model) Load(path string) error

func (*Model) Save

func (m *Model) Save(path string) error

type Tokenizer

type Tokenizer struct {
	IDToToken []string
	TokenToID map[string]int
	// Merges are (leftID, rightID) in priority order; first merge has highest priority.
	Merges [][2]int
}

Tokenizer is a byte-level BPE trained from a corpus.

func TrainBPE

func TrainBPE(text string, vocabSize int, progress ...BPEProgress) *Tokenizer

TrainBPE builds a vocabulary of about vocabSize tokens from text.

func TrainBPEKeep

func TrainBPEKeep(text string, vocabSize, minTokens int, progress ...BPEProgress) *Tokenizer

TrainBPEKeep is TrainBPE that stops before the corpus shrinks below minTokens. minTokens 0 merges until vocabSize or no pair still repeats.

func (*Tokenizer) Decode

func (t *Tokenizer) Decode(ids []int) string

Decode maps token IDs back to text (skips specials).

func (*Tokenizer) Encode

func (t *Tokenizer) Encode(text string, progress ...EncodeProgress) []int

Encode maps text to token IDs using learned merges.

func (*Tokenizer) Load

func (t *Tokenizer) Load(path string) error

func (*Tokenizer) Save

func (t *Tokenizer) Save(path string) error

func (*Tokenizer) VocabSize

func (t *Tokenizer) VocabSize() int

type TrainOpts

type TrainOpts struct {
	CorpusPath string
	CorpusText string // if set, used instead of reading CorpusPath
	OutDir     string
	VocabSize  int
	LR         float32
	Epochs     int // training steps (batches), not full data passes
	EvalEvery  int
	SaveEvery  int
	SampleLen  int
	ValFrac    float64
	Resume     bool
	Toy        bool
	Optim      string // "sgd" (default), "adam", or "adamw"; empty on resume uses the checkpoint
	Config     Config // zero VocabSize filled after tokenizer
}

TrainOpts controls a training run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL