fileiterator

package
v0.0.0-...-7e3c669 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 7, 2026 License: MIT Imports: 33 Imported by: 0

README

File Iterator Package

The fileiterator package provides high-level iterators for structured file formats (JSONL, CSV) with automatic compression detection.

Features

  • Binary record iterators for fixed-size records with compression support
  • File loaders with automatic compression detection
  • JSONL (JSON Lines) support with typed and untyped parsing
  • CSV support with flexible options and map-based iteration
  • Automatic compression detection (7 formats: .gz, .zst, .zlib, .lz4, .br, .xz, plain)
  • URL support - works with both local files and HTTP/HTTPS URLs
  • Error handling with detailed error messages (line/row numbers)
  • Progress tracking - prints row/line counts
  • Streaming processing - low memory usage for large files

File Loaders

Auto-Detection Loaders
FUOpen - Open with Auto-Decompression

Opens a file or URL and returns an io.ReadCloser with automatic decompression.

Supports 7 compression formats:

  • Gzip (.gz)
  • Zstd (.zst)
  • Zlib (.zlib, .zz)
  • LZ4 (.lz4)
  • Brotli (.br)
  • XZ (.xz)
  • Plain files
import "github.com/parf/homebase-go-lib/fileiterator"

// Automatically decompresses based on file extension
r := fileiterator.FUOpen("data.txt.gz")
defer r.Close()
data, _ := io.ReadAll(r)
LoadBinFile - Load Binary File

Loads a file into a byte buffer with automatic decompression:

var data []byte
fileiterator.LoadBinFile("data.bin.gz", &data)
fmt.Printf("Loaded %d bytes\n", len(data))
IterateLines - Process Lines

Process lines in a text file with automatic decompression:

fileiterator.IterateLines("log.txt.zst", func(line string) {
    fmt.Println(line)
})
IterateIDTabFile - Process Tab-Separated ID-Name Pairs

Process tab-separated files where IDs are hexadecimal int32 and names are lowercased:

fileiterator.IterateIDTabFile("ids.tab.gz", func(id int32, name string) {
    fmt.Printf("ID: %x, Name: %s\n", id, name)
})

Binary Record Iterators

IterateBinaryRecords - Auto-Detection

Iterate over fixed-size binary records with automatic compression detection:

// Detects .gz, .zst, .zlib/.zz extensions automatically
fileiterator.IterateBinaryRecords("records.bin.gz", 64, func(record []byte) {
    // Process each 64-byte record
})

// Plain binary files also supported
fileiterator.IterateBinaryRecords("records.bin", 64, func(record []byte) {
    // Process uncompressed records
})
Explicit Format Iterators

For explicit compression format control:

// Gzip
fileiterator.IterateGzipRecords("data.bin.gz", 64, processor)

// Zstd
fileiterator.IterateZstdRecords("data.bin.zst", 64, processor)

// Zlib (RFC 1950)
fileiterator.IterateZlibRecords("data.bin.zlib", 64, processor)

Use cases:

  • Fixed-size binary records (database dumps, network packets, etc.)
  • Streaming processing of large binary files
  • Works with compressed and uncompressed files

JSONL (JSON Lines) Support

IterateJSONL - Untyped

Process JSONL files with generic map[string]any objects:

import "github.com/parf/homebase-go-lib/fileiterator"

err := fileiterator.IterateJSONL("data.jsonl.gz", func(obj map[string]any) error {
    fmt.Printf("ID: %v, Name: %v\n", obj["id"], obj["name"])
    return nil
})
IterateJSONLTyped - Type-Safe

Process JSONL files with strongly-typed structs:

type User struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
    Email string `json:"email"`
}

err := fileiterator.IterateJSONLTyped("users.jsonl", func(user User) error {
    fmt.Printf("User: %s <%s>\n", user.Name, user.Email)
    return nil
})

Supports all 7 compression formats:

  • Plain: data.jsonl
  • Gzip: data.jsonl.gz
  • Zstd: data.jsonl.zst
  • Zlib: data.jsonl.zlib or .zz
  • LZ4: data.jsonl.lz4
  • Brotli: data.jsonl.br
  • XZ: data.jsonl.xz
  • URLs: http://example.com/data.jsonl.gz (any format)

CSV Support

CSVOptions

Configure CSV parsing behavior:

opts := fileiterator.DefaultCSVOptions()
opts.Comma = '\t'              // Tab-separated (TSV)
opts.SkipHeader = true          // Skip first row
opts.TrimLeadingSpace = true    // Trim spaces
opts.Comment = '#'              // Comment character
IterateCSV - Array-based

Process CSV files row by row as string slices:

opts := fileiterator.DefaultCSVOptions()
opts.SkipHeader = true

err := fileiterator.IterateCSV("data.csv.gz", opts, func(row []string) error {
    fmt.Printf("Name: %s, Age: %s\n", row[0], row[1])
    return nil
})
IterateCSVMap - Map-based

Process CSV files with header row, get each row as a map:

err := fileiterator.IterateCSVMap("users.csv", fileiterator.DefaultCSVOptions(), func(row map[string]string) error {
    fmt.Printf("Name: %s, Email: %s\n", row["name"], row["email"])
    return nil
})

Supports all 7 compression formats:

  • Plain: data.csv
  • Gzip: data.csv.gz
  • Zstd: data.csv.zst
  • Zlib: data.csv.zlib or .zz
  • LZ4: data.csv.lz4
  • Brotli: data.csv.br
  • XZ: data.csv.xz
  • URLs: http://example.com/data.csv.gz (any format)
  • TSV (tab-separated) - set Comma to '\t'
  • Custom delimiters (pipe, semicolon, etc.)

Examples

JSONL from URL
err := fileiterator.IterateJSONL("https://example.com/logs.jsonl.gz", func(log map[string]any) error {
    fmt.Printf("Timestamp: %v, Level: %v\n", log["timestamp"], log["level"])
    return nil
})
CSV with Custom Delimiter
opts := fileiterator.DefaultCSVOptions()
opts.Comma = '|'  // Pipe-separated
opts.SkipHeader = true

err := fileiterator.IterateCSV("data.psv", opts, func(row []string) error {
    // Process pipe-separated values
    return nil
})
Typed JSONL Processing
type LogEntry struct {
    Timestamp string `json:"timestamp"`
    Level     string `json:"level"`
    Message   string `json:"message"`
}

err := fileiterator.IterateJSONLTyped("logs.jsonl.zst", func(entry LogEntry) error {
    if entry.Level == "ERROR" {
        fmt.Printf("[%s] %s\n", entry.Timestamp, entry.Message)
    }
    return nil
})
CSV Map with Progress
count := 0
err := fileiterator.IterateCSVMap("users.csv.gz", fileiterator.DefaultCSVOptions(), func(row map[string]string) error {
    count++
    if count%1000 == 0 {
        fmt.Printf("Processed %d users...\n", count)
    }
    return nil
})
fmt.Printf("Total users: %d\n", count)

Error Handling

All iterator functions return detailed errors with line/row numbers:

err := fileiterator.IterateJSONL("data.jsonl", func(obj map[string]any) error {
    if obj["id"] == nil {
        return fmt.Errorf("missing required field: id")
    }
    return nil
})

if err != nil {
    // Error format: "line 42: processor error: missing required field: id"
    log.Fatal(err)
}

Performance

  • Streaming processing - low memory footprint
  • Automatic buffering for optimal performance
  • Progress messages to stdout
  • Suitable for large files (GB+)

Compression Support

All functions automatically detect compression by file extension.

7 formats supported:

  • Gzip (.gz) - Standard gzip compression (RFC 1952)
  • Zstd (.zst) - Modern, faster compression
  • Zlib (.zlib, .zz) - Zlib compression (RFC 1950)
  • LZ4 (.lz4) - Fast compression algorithm
  • Brotli (.br) - Modern web compression
  • XZ (.xz) - High compression ratio
  • Plain files - No compression

No special code needed - just use compressed files directly. The library automatically detects the format and decompresses on-the-fly.

URL Support

All functions work with HTTP/HTTPS URLs:

  • Automatically fetches and streams content
  • Works with compressed URLs
  • No temporary files created

When to Use

Use this package when:

  • Processing structured data files (JSONL, CSV)
  • Files can be large (streaming processing)
  • Need automatic compression handling
  • Want type-safe JSONL parsing
  • Processing data from URLs

Use other packages when:

  • Need random access to file contents
  • Need SQL database operations (use sql package)
  • Custom parsing requirements

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FUCreate 

func FUCreate(filename string) io.WriteCloser

FUCreate creates a file and returns an io.WriteCloser. Automatically compresses based on file extension: .gz (gzip), .zst (zstd default), .zst1 (zstd level 1), .zst2 (zstd level 2), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func FUOpen 

func FUOpen(file_or_url string) io.ReadCloser

FUOpen opens a file or URL and returns an io.ReadCloser. Automatically detects and decompresses files based on extension: .gz (gzip), .zst/.zst1/.zst2 (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func IterateBinaryRecords 

func IterateBinaryRecords(filename string, recordSize int, processor func([]byte))

IterateBinaryRecords iterates over a file of binary records of fixed recordSize. Automatically detects compression format by extension: .gz (gzip), .zst (zstd), .zlib/.zz (zlib) If no compression extension is detected, processes file as plain binary. Calls processor function on every record.

filename - "filename" or "http://url" recordSize - size of each binary record in bytes

func IterateCSV 

func IterateCSV(filename string, opts CSVOptions, processor func([]string) error) error

IterateCSV processes a CSV file row by row. Supports compression auto-detection by extension (.gz, .zst). Each row is passed to the processor function as a slice of strings.

filename - "filename" or "http://url" (with optional .gz or .zst extension) opts - CSV parsing options (use DefaultCSVOptions() for defaults) processor - function that receives each CSV row

Example: 

fileiterator.IterateCSV("data.csv.gz", fileiterator.DefaultCSVOptions(), func(row []string) error {
    fmt.Printf("Row: %v\n", row)
    return nil
})

func IterateCSVMap 

func IterateCSVMap(filename string, opts CSVOptions, processor func(map[string]string) error) error

IterateCSVMap processes a CSV file with header row, returning each row as a map. Supports compression auto-detection by extension (.gz, .zst). First row is used as headers (keys), subsequent rows are returned as map[header]value.

filename - "filename" or "http://url" (with optional .gz or .zst extension) opts - CSV parsing options (SkipHeader is ignored, first row is always header) processor - function that receives each CSV row as a map

Example: 

fileiterator.IterateCSVMap("users.csv", fileiterator.DefaultCSVOptions(), func(row map[string]string) error {
    fmt.Printf("Name: %s, Email: %s\n", row["name"], row["email"])
    return nil
})

func IterateFlatBufferList 

func IterateFlatBufferList(filename string, processor func([]byte) error) error

IterateFlatBufferList iterates over a FlatBuffer file containing multiple records Each record should be prefixed with a 4-byte length (uint32, little-endian) Supports compressed files via FUOpen auto-detection (.fb, .fb.gz, .fb.zst, .fb.lz4, etc.)

File format: [length1:uint32][record1:bytes][length2:uint32][record2:bytes]...

func IterateGzipRecords 

func IterateGzipRecords(filename string, recordSize int, processor func([]byte))

IterateGzipRecords iterates over gzip-compressed file of binary records (explicit gzip)

func IterateIDTabFile 

func IterateIDTabFile(filename string, processor func(int32, string))

IterateIDTabFile iterates over TAB separated (ID <tab> NAME) file with automatic decompression ID is parsed as hexadecimal int32, NAME is converted to lowercase Supported: .gz (gzip), .zst (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func IterateJSONL 

func IterateJSONL(filename string, processor func(map[string]any) error) error

IterateJSONL processes a JSONL (JSON Lines) file line by line. Supports compression auto-detection by extension (.gz, .zst). Each line is parsed as JSON and passed to the processor function.

filename - "filename" or "http://url" (with optional .gz or .zst extension) processor - function that receives each parsed JSON object

Example: 

fileiterator.IterateJSONL("data.jsonl.gz", func(obj map[string]any) error {
    fmt.Printf("ID: %v, Name: %v\n", obj["id"], obj["name"])
    return nil
})

func IterateJSONLTyped 

func IterateJSONLTyped[T any](filename string, processor func(T) error) error

IterateJSONLTyped processes a JSONL file with a typed struct. Supports compression auto-detection by extension (.gz, .zst). Each line is parsed into the provided type T.

filename - "filename" or "http://url" (with optional .gz or .zst extension) processor - function that receives each parsed object of type T

Example: 

type User struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
}
fileiterator.IterateJSONLTyped("users.jsonl", func(user User) error {
    fmt.Printf("User: %s\n", user.Name)
    return nil
})

func IterateLines 

func IterateLines(filename string, processor func(string))

IterateLines processes lines in a file with automatic decompression Supported: .gz (gzip), .zst (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func IterateMsgPack 

func IterateMsgPack(filename string, processor func(any) error) error

IterateMsgPack iterates over a MessagePack stream file Each record in the file is decoded and passed to the processor Supports compressed files via FUOpen auto-detection

func IterateMsgPackTyped 

func IterateMsgPackTyped[T any](filename string, processor func(T) error) error

IterateMsgPackTyped iterates over a MessagePack stream file with type safety Each record is decoded into type T and passed to the processor Supports compressed files via FUOpen auto-detection

func IterateParquet 

func IterateParquet(filename string, processor func(ParquetRecord) error) error

IterateParquet reads Parquet file and calls processor for each record Automatically handles compression detection via FUOpen Schema: id, name, email, age, score, active, category, timestamp

func IterateParquetAny 

func IterateParquetAny(filename string, processor func(map[string]any) error) error

IterateParquetAny reads Parquet file and calls processor for each record as map[string]any Automatically handles compression detection via file extension Supports ANY Parquet schema

func IterateZlibRecords 

func IterateZlibRecords(filename string, recordSize int, processor func([]byte))

IterateZlibRecords iterates over zlib-compressed file of binary records (explicit zlib) This is the original function for backward compatibility

func IterateZstdRecords 

func IterateZstdRecords(filename string, recordSize int, processor func([]byte))

IterateZstdRecords iterates over zstd-compressed file of binary records (explicit zstd)

func LoadBinFile 

func LoadBinFile(filename string, dest *[]byte)

LoadBinFile loads a file with automatic decompression into a byte buffer Supported: .gz (gzip), .zst (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func LoadFlatBuffer 

func LoadFlatBuffer(filename string) ([]byte, error)

LoadFlatBuffer loads a FlatBuffer from a file with buffered IO Returns the raw bytes for zero-copy access Uses 4MB buffer for maximum performance

func LoadFlatBufferCompressed 

func LoadFlatBufferCompressed(filename string) ([]byte, error)

LoadFlatBufferCompressed loads a FlatBuffer from a compressed file Supports all compression formats via FUOpen auto-detection Compression formats: .gz, .zst, .zst1, .zst2, .zlib, .zz, .lz4, .br, .xz

func LoadMmap 

func LoadMmap(filename string) ([]byte, error)

LoadMmap loads a file using memory mapping for ultra-fast access Returns the data as a byte slice. The underlying memory mapping is managed internally and will be cleaned up when appropriate.

WARNING: The returned slice is backed by mmap. Do not use it after the program terminates or if you need guaranteed persistence across process restarts. For long-lived data that outlives the file access, copy the bytes.

Best for: Large read-heavy files where random access is needed Not suitable for: Compressed files, streaming, or write operations

func LoadMsgPack 

func LoadMsgPack(filename string, dest any) error

LoadMsgPack loads data from a MessagePack file with buffered IO Uses 4MB buffer for maximum performance

func LoadMsgPackCompressed 

func LoadMsgPackCompressed(filename string, dest any) error

LoadMsgPackCompressed loads data from a compressed MessagePack file Supports all compression formats via FUOpen auto-detection

func LoadMsgPackMap 

func LoadMsgPackMap(filename string) (map[string]any, error)

LoadMsgPackMap loads a map from MessagePack file with buffered IO

func LoadMsgPackMapCompressed 

func LoadMsgPackMapCompressed(filename string) (map[string]any, error)

LoadMsgPackMapCompressed loads a map from compressed MessagePack file

func ReadInput 

func ReadInput(filename string) ([]map[string]any, error)

ReadInput reads any supported format and returns generic records

func ReadSQLInput 

func ReadSQLInput(driver, dsn, query string) ([]map[string]any, error)

ReadSQLInput executes a SQL query and returns generic records

func SaveFlatBuffer 

func SaveFlatBuffer(filename string, builder *flatbuffers.Builder) error

SaveFlatBuffer saves a FlatBuffer to a file with buffered IO Uses 4MB buffer for maximum performance

func SaveFlatBufferCompressed 

func SaveFlatBufferCompressed(filename string, builder *flatbuffers.Builder) error

SaveFlatBufferCompressed saves a FlatBuffer to a compressed file Supports all compression formats via file extension: .gz (gzip), .zst (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz)

func SaveFlatBufferList 

func SaveFlatBufferList(filename string, records [][]byte) error

SaveFlatBufferList saves multiple FlatBuffer records to a file with length prefixes Each record is prefixed with a 4-byte length (uint32, little-endian) Supports compression via file extension (.fb.gz, .fb.zst, .fb.lz4, etc.)

func SaveMsgPack 

func SaveMsgPack(filename string, data any) error

SaveMsgPack saves data to a MessagePack file with buffered IO Uses 4MB buffer for maximum performance

func SaveMsgPackCompressed 

func SaveMsgPackCompressed(filename string, data any) error

SaveMsgPackCompressed saves data to a compressed MessagePack file Supports all compression formats via file extension: .gz (gzip), .zst (zstd), .zlib/.zz (zlib), .lz4 (lz4), .br (brotli), .xz (xz) Common usage: filename.msgpack.zst (MessagePack + Zstandard)

func SaveMsgPackMap 

func SaveMsgPackMap(filename string, data map[string]any) error

SaveMsgPackMap saves a map to MessagePack file with buffered IO

func SaveMsgPackMapCompressed 

func SaveMsgPackMapCompressed(filename string, data map[string]any) error

SaveMsgPackMapCompressed saves a map to compressed MessagePack file Common usage: data.msgpack.zst (MessagePack + Zstandard)

func WriteOutput 

func WriteOutput(filename string, records []map[string]any) error

WriteOutput writes records to any supported format

func WriteParquet 

func WriteParquet(filename string, records []ParquetRecord) error

WriteParquet writes records to Parquet file with Snappy compression Automatically handles additional compression via FUCreate (if filename has .gz/.zst/.lz4 extension) Schema: id, name, email, age, score, active, category, timestamp

func WriteParquetAny 

func WriteParquetAny(filename string, records []map[string]any) error

WriteParquetAny writes generic records to Parquet file with Snappy compression Automatically infers schema from data - supports ANY record structure Handles compression via FUCreate (if filename has .gz/.zst/.lz4 extension) Supported types: int64, float64, string, bool

Types

type CSVOptions 

type CSVOptions struct {
	Comma            rune // Field delimiter (default: ',')
	Comment          rune // Comment character (default: 0, disabled)
	SkipHeader       bool // Skip first row as header
	TrimLeadingSpace bool // Trim leading space in fields
}

CSVOptions configures CSV parsing behavior

func DefaultCSVOptions 

func DefaultCSVOptions() CSVOptions

DefaultCSVOptions returns default CSV parsing options

type MmapFile 

type MmapFile struct {
	Data []byte
	// contains filtered or unexported fields
}

MmapFile represents a memory-mapped file with lifecycle management

func MmapOpen 

func MmapOpen(filename string) (*MmapFile, error)

MmapOpen opens a file with memory mapping for ultra-fast read access Returns MmapFile which must be closed by the caller Best for: Large read-heavy files where random access is needed Not suitable for: Compressed files, streaming, or write operations

func (*MmapFile) Close 

func (m *MmapFile) Close() error

Close unmaps the memory and closes the file

type ParquetRecord 

type ParquetRecord struct {
	ID        int64
	Name      string
	Email     string
	Age       int64
	Score     float64
	Active    bool
	Category  string
	Timestamp int64
}

ParquetRecord represents a generic record for Parquet operations Standard schema: id, name, email, age, score, active, category, timestamp

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.