Documentation
¶
Index ¶
Constants ¶
const ( TokenPAD = 0 TokenUNK = 1 TokenBOS = 2 TokenEOS = 3 )
Variables ¶
This section is empty.
Functions ¶
func Learn ¶
Learn is the CLI entry for from-scratch GPT training.
learn [toy|path] [resume] [sgd|adam|adamw] learn train on The War of the Worlds with SGD learn toy overfit a tiny repeated corpus learn <path> train on a custom text file learn resume resume from data/checkpoints/learn learn toy resume resume toy run from data/checkpoints/learn-toy learn toy adam same run with Adam (default lr 1e-3) learn toy adamw AdamW, weight decay 0.01, same default lr
func NextTokenLoss ¶
NextTokenLoss is mean cross-entropy over all positions. logits [B,T,V], targets [B][T] class indexes (next token at each position).
Types ¶
type BPEProgress ¶
type BPEProgress func(vocabNow, vocabTarget int)
BPEProgress is called during TrainBPE with current/target vocab size.
type Config ¶
type Config struct {
VocabSize int
ContextLen int // T
EmbedSize int // D
Layers int
Heads int
MLPSize int
BatchSize int
Eps float32
}
Config holds hyperparameters for a small GPT-style decoder.
func DefaultConfig ¶
DefaultConfig returns small training defaults.
type EncodeProgress ¶
type EncodeProgress func(mergesDone, mergesTotal int)
EncodeProgress reports encoding progress (merges applied / total merges).
type Model ¶
type Model struct {
Cfg Config
TokEmb *neural.Embeddings
PosEmb *llm.PositionEmbeddings
Blocks []*Block
LN *neural.LayerNorm
LMHead *neural.Linear
}
Model is a GPT-style decoder-only transformer for next-token training.
type Tokenizer ¶
type Tokenizer struct {
IDToToken []string
TokenToID map[string]int
// Merges are (leftID, rightID) in priority order; first merge has highest priority.
Merges [][2]int
}
Tokenizer is a byte-level BPE trained from a corpus.
func TrainBPE ¶
func TrainBPE(text string, vocabSize int, progress ...BPEProgress) *Tokenizer
TrainBPE builds a vocabulary of about vocabSize tokens from text.
func TrainBPEKeep ¶
func TrainBPEKeep(text string, vocabSize, minTokens int, progress ...BPEProgress) *Tokenizer
TrainBPEKeep is TrainBPE that stops before the corpus shrinks below minTokens. minTokens 0 merges until vocabSize or no pair still repeats.
type TrainOpts ¶
type TrainOpts struct {
CorpusPath string
CorpusText string // if set, used instead of reading CorpusPath
OutDir string
VocabSize int
LR float32
Epochs int // training steps (batches), not full data passes
EvalEvery int
SaveEvery int
SampleLen int
ValFrac float64
Resume bool
Toy bool
Optim string // "sgd" (default), "adam", or "adamw"; empty on resume uses the checkpoint
Config Config // zero VocabSize filled after tokenizer
}
TrainOpts controls a training run.