tokenizer

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 18, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Overview

Package tokenizer splits command-line input into typed tokens and provides scanners for traversing them.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner provides a forward-only, peekable cursor over a sequence of tokens.

func NewScanner

func NewScanner(tokens []Token) *Scanner

NewScanner initializes a new Scanner with the provided tokens.

func (*Scanner) HasNext

func (s *Scanner) HasNext() bool

HasNext returns true if there are more tokens remaining to be read.

func (*Scanner) Next

func (s *Scanner) Next() (Token, bool)

Next consumes and returns the next token, or returns (Token{}, false) if exhausted.

func (*Scanner) Peek

func (s *Scanner) Peek() (Token, bool)

Peek returns the next token without advancing the cursor, or returns (Token{}, false) if exhausted.

type Token

type Token struct {
	// Type is the classification of this token.
	Type TokenType

	// Raw is the original, unmodified argument string as provided in os.Args.
	Raw string

	// Name is the parsed name for long flags (e.g. "port" for "--port=8080").
	// For single short flags, it is the string representation of the rune (e.g. "v" for "-v").
	Name string

	// Value contains the attached value if provided with "=" or attached to a short flag.
	// For example, in "--port=8080" Value is "8080". In "-p8080" Value is "8080".
	Value string

	// HasAttachedValue indicates if a value was directly attached via "=" or short flag suffix.
	HasAttachedValue bool

	// ShortRunes contains all runes in a short flag cluster (e.g. ['x', 'z', 'v', 'f'] for "-xzvf").
	ShortRunes []rune

	// Index is the zero-based index of this token in the input args slice.
	Index int
}

Token represents a single parsed syntactic unit from os.Args.

func Tokenize

func Tokenize(args []string) []Token

Tokenize converts a slice of command-line arguments (typically os.Args[1:]) into a slice of structured Token objects conforming to POSIX and GNU CLI conventions.

func (Token) CanBindAsFlagValue added in v0.6.0

func (t Token) CanBindAsFlagValue() bool

CanBindAsFlagValue reports whether the token may be consumed as the next value of a flag. A flag-shaped token (long flag, short flag, or the "--" terminator) never binds as a value unless it is a negative number, which the parser treats as an operand. The runtime parser and the completion walker share this rule so their value attribution cannot drift.

func (Token) IsNegativeNumber

func (t Token) IsNegativeNumber() bool

IsNegativeNumber checks if a token's raw representation corresponds to a numeric literal (e.g. "-10", "-3.14", "-1e-3").

func (Token) String

func (t Token) String() string

String provides a formatted representation of the token for debugging.

type TokenType

type TokenType int

TokenType represents the syntactic category of a command-line token.

const (
	// TokenTypeUnknown represents an unclassified token.
	TokenTypeUnknown TokenType = iota

	// TokenTypePositional represents a positional argument / operand (e.g., "build", "file.txt", or "-" for stdin).
	TokenTypePositional

	// TokenTypeLongFlag represents a long option starting with "--" (e.g., "--verbose" or "--port=8080").
	TokenTypeLongFlag

	// TokenTypeShortFlag represents one or more short options starting with "-" (e.g., "-v", "-xzvf", or "-p8080").
	TokenTypeShortFlag

	// TokenTypeTerminator represents the explicit "--" option terminator.
	TokenTypeTerminator
)

func (TokenType) String

func (t TokenType) String() string

String returns a human-readable name for the token type.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL