Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Block ¶
type Block struct {
D int
LN1 *neural.LayerNorm // layer norm
Attn *llm.Attention // self-attention
LN2 *neural.LayerNorm // layer norm
MLP *MLP // Multi-Layer Perceptron
}
Transformer block
func (*Block) ForwardNext ¶ added in v1.1.1
type MLP ¶
GPT-2 feed-forward MLP f(x) = linear( GeLU( linear(x) ) ) x → Linear(D→4D) → GELU → Linear(4D→D) First linear == CFC Second linear = CProj
type Model ¶
type Model struct {
VocabSize int
EmbeddingSize int
ContextSize int
Heads int
Layers int
Eps float32
DropP float32
WTE *neural.Embeddings // token embedding
WPE *llm.PositionEmbeddings // position embedding
Blocks []*Block // transformer blocks
LNF *neural.LayerNorm // final layer norm
LMHead *neural.Linear // projection to vocab
}
func NewModel ¶
func NewModel(p *ModelParams) *Model
func (*Model) ForwardNext ¶ added in v1.1.1
type ModelParams ¶
type Tokenizer ¶
type Tokenizer struct {
// token string -> id
Vocab map[string]int
// id -> token string
InvVocab []string
// contains filtered or unexported fields
}
Tokenizer supports decoding token IDs using vocab.json. It implements the GPT-2 byte decoder so you get normal UTF-8 text out.
func NewTokenizer ¶
func (*Tokenizer) DebugSample ¶
Optional helper: print a few tokens for debugging.
func (*Tokenizer) Decode ¶
Decode converts token ids into text (UTF-8), by concatenating decoded tokens.
Source Files
Click to show internal directories.
Click to hide internal directories.