go-xberg-sdk

go-xberg-sdk

English | 简体中文

A lightweight Go document extraction SDK built on the Xberg Go binding v1.0.12, with a CloudWeGo Eino Parser adapter. It converts PDFs, Office documents, images, email, web content, and structured text into Markdown, pages, chunks, or semantic elements suitable for RAG pipelines.

Packages

• github.com/sungithubid/go-xberg-sdk/xberg is the framework-independent core SDK. It accepts file paths, []byte, and io.Reader inputs.
• github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg implements Eino's parser.Parser and supports Document, Page, Chunk, and Element output modes.

go-xberg-sdk/
├── xberg/                              # Core SDK
├── components/document/parser/xberg/   # Eino Parser adapter
├── examples/                           # Core, Eino, OCR, and LLM extraction/vision examples
├── scripts/                            # Native dependency and bundle tooling
├── docs/NATIVE_BUNDLE.md               # Runtime bundle guide
└── .github/workflows/                   # CI and multi-platform releases

Distribution model

The project separates Go source distribution from native runtime delivery:

1. The Go module stays lightweight. Git tags and module zip files contain Go source, documentation, and scripts, but not large platform-specific native libraries.
2. GitHub Releases provide native bundles. Each release downloads Xberg's checksummed official v1.0.12 Go FFI, then publishes archives and SHA-256 sidecars for the three supported OS/architecture combinations. A bundle contains libxberg_ffi, recursively discovered non-system sidecars, relative runtime paths, and licenses.

Consequently, go get does not install the native runtime automatically. Xberg v1.0.12's official FFI enables default -> full -> formats -> excel, so XLSX and the other full document extractors are available without a local Rust build. This project's Release bundle also carries the non-system dynamic libraries referenced by the upstream asset.

Requirements

• Go 1.26 or later.
• CGO_ENABLED=1 and a C compiler.
• The Go binding and native FFI must use the same Xberg version. This project currently pins both to v1.0.12.
• Building a Linux release bundle requires patchelf. macOS uses otool and install_name_tool from the system toolchain.
• Current project releases support macOS ARM64, Linux AMD64, and Linux ARM64.
• The upstream v1.0.12 full FFI dynamically references libheif, and Linux also ships ONNX Runtime. The project Release workflow collects those non-system runtime dependencies into the bundle.

Installation

Install the Go packages

go get github.com/sungithubid/go-xberg-sdk/xberg

# For the Eino adapter:
go get github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg

Prepare the native FFI for development

Run Xberg's official setup command in your application module. It downloads and verifies the native library matching the Go binding and generates a local link shim for the application:

go run github.com/xberg-io/xberg/packages/go/cmd/setup@v1.0.12

This downloads the upstream default/full FFI and verifies its SHA-256 sidecar. It supports XLSX. On macOS, install libheif (brew install libheif) if the upstream library's absolute Homebrew dependency is not already present. For deployment, prefer this project's matching Release bundle, whose dependency paths are made relative.

When developing this repository directly, use:

make native
make test

To replace any stale native library and verify XLSX/PDF support locally:

make native-refresh
make test

No Rust toolchain or Xberg source build is required. make test always runs a minimal XLSX integration test, so a future native feature regression fails CI. Set XBERG_REAL_XLSX=/path/to/workbook.xlsx to include a real workbook in the same test run. Set XBERG_OCR_ENABLED=1 to run the generated-image OCR test; its first run may populate Xberg's tessdata cache.

If you previously used the retired local Rust source-build flow, remove only the files it created with:

make clean-source-build
rm -f /tmp/xberg-full-build.log

# Only uninstall formulas installed solely for the retired source build.
brew uninstall boost cmake libmagic pkg-config

# Only if rustup was installed solely for this build.
rustup self uninstall

The earlier source "$HOME/.cargo/env" command changed only that shell and has nothing to uninstall. Keep libheif: the official v1.0.12 macOS FFI uses it at development time. Do not remove pre-existing Tesseract packages unless you have confirmed no other application uses them.

Use a GitHub Release bundle

For application distribution, download the asset matching the go-xberg-sdk version and target platform:

VERSION=vX.Y.Z
PLATFORM=linux-x86_64

gh release download "$VERSION" \
  --repo sungithubid/go-xberg-sdk \
  --pattern "go-xberg-sdk-native-${PLATFORM}.tar.gz*"
sha256sum -c "go-xberg-sdk-native-${PLATFORM}.tar.gz.sha256"
tar -xzf "go-xberg-sdk-native-${PLATFORM}.tar.gz"

On macOS, use shasum -a 256 -c <checksum-file>. Keep every .so or .dylib from the extracted archive in the same directory. The bundle's own README.md contains detailed deployment instructions.

Point CGO at the extracted directory when building:

CGO_ENABLED=1 \
CGO_LDFLAGS="-L/absolute/path/to/go-xberg-sdk-native-linux-x86_64" \
go build ./...

For local execution, also set LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on macOS. A distributed application can instead place all bundle libraries next to its executable and configure a $ORIGIN or @loader_path runtime search path.

Core SDK quick start

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/sungithubid/go-xberg-sdk/xberg"
)

func main() {
    extractor, err := xberg.New(nil)
    if err != nil {
        log.Fatal(err)
    }

    result, err := extractor.ExtractFile(context.Background(), "manual.docx")
    if err != nil {
        log.Fatal(err)
    }

    for _, document := range result.Results {
        fmt.Printf("MIME: %s\n%s\n", document.MimeType, document.Content)
    }
}

Extractor owns no native handle that requires cleanup, so it has no Close method. It is safe for concurrent use.

In addition to ExtractFile, the SDK supports byte slices and readers:

result, err := extractor.ExtractBytes(ctx, data,
    xberg.WithFilename("manual.docx"),
)

result, err = extractor.ExtractReader(ctx, reader,
    xberg.WithFilename("manual.pdf"),
    xberg.WithMIMEType("application/pdf"),
)

For ZIP-container formats such as DOCX, XLSX, and PPTX, always provide a filename or MIME hint when using in-memory or reader input.

Eino Parser quick start

package main

import (
    "context"
    "fmt"
    "os"

    einoparser "github.com/cloudwego/eino/components/document/parser"
    xbergparser "github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg"
)

func main() {
    ctx := context.Background()
    p, err := xbergparser.NewParser(ctx, &xbergparser.Config{
        DocumentMode: xbergparser.DocumentModeChunk,
    })
    if err != nil {
        panic(err)
    }

    file, err := os.Open("manual.pdf")
    if err != nil {
        panic(err)
    }
    defer file.Close()

    docs, err := p.Parse(ctx, file,
        einoparser.WithURI("manual.pdf"),
        einoparser.WithExtraMeta(map[string]any{"knowledge_base": "manuals"}),
    )
    if err != nil {
        panic(err)
    }

    for _, doc := range docs {
        fmt.Printf("%s: %s\n", doc.ID, doc.Content)
    }
}

See the Eino Parser Chinese guide for output modes, OCR, PDF, chunking, and metadata options. Advanced Xberg per-file settings remain accessible through WithNativeFileConfig.

Examples and verification

make test
make test-race
make vet
make example-core
make example-eino
make example-ocr OCR_INPUT=/path/to/scan.png

# 1. LLM structured metadata extraction (OpenAI-compatible API extracting title, summary, tags, entities, etc.)
export OPENAI_API_KEY="sk-..."
# export OPENAI_BASE_URL="https://api.deepseek.com/v1" # Optional OpenAI-compatible endpoint
# export OPENAI_MODEL="gpt-4o-mini"                   # Optional model
make example-llm-structured LLM_INPUT=/path/to/document.pdf

# 2. Vision/Multimodal LLM high-fidelity OCR (GPT-4o, Qwen-VL, GLM-4V, etc.)
# export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
# export OPENAI_MODEL="qwen-vl-max"
make example-llm-vision VISION_INPUT=/path/to/image-or-scanned.pdf

The OCR example deliberately has no repository-local input fixture. Supply an image or scanned PDF explicitly. LLM examples read standard OPENAI_API_KEY, OPENAI_BASE_URL, and OPENAI_MODEL environment variables, supporting OpenAI, DeepSeek, Qwen, Kimi, Ollama, and other compatible providers.

Build a native Release

Pushing a v* tag triggers the release-native workflow. On all three native runners, it:

1. Downloads Xberg's official checksummed v1.0.12 Go FFI and installs only the platform runtime/packaging dependencies.
2. Runs the Go integration test containing a real minimal XLSX workbook.
3. Recursively collects non-system dynamic dependencies and discoverable licenses.
4. Rewrites dependency paths to @loader_path on macOS or $ORIGIN on Linux.
5. Links and tests against the packaged directory.
6. Generates and uploads dist/*.tar.gz and matching .sha256 files.

For a manual local package, run make release-native VERSION=<tag>. The Makefile downloads the pinned official FFI, records its version/feature provenance, and tests the rewritten bundle.

Runtime boundaries

The native bundle removes implicit dependencies on build-host paths such as a Homebrew prefix, but it is not a fully static environment. It still relies on the target OS loader, base system libraries, and a compatible glibc or macOS version. Optional OCR, layout, embedding, and transcription features can also require model assets or network downloads. Test those capabilities in an offline environment matching production before deployment.

The underlying Xberg FFI call is synchronous. A context.Context deadline is converted into an Xberg timeout, and cancellation is checked before and after the native call, but cancellation cannot guarantee an immediate interruption after execution has entered native code.

License

This project is licensed under the MIT License. Licenses for Xberg, ONNX Runtime, and discoverable native sidecars are included under third_party/ and in each bundle's licenses/ directory. Review all native codec licenses before choosing a distribution model for your product.