go-xberg-sdk

go-xberg-sdk

English | 简体中文

A lightweight Go document extraction SDK built on the Xberg Go binding v1.0.14, with a CloudWeGo Eino Parser adapter. It converts PDFs, Office documents, images, email, web content, and structured text into Markdown, pages, chunks, or semantic elements suitable for RAG pipelines.

Packages

• github.com/sungithubid/go-xberg-sdk/xberg is the framework-independent core SDK. It accepts file paths, []byte, and io.Reader inputs.
• github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg implements Eino's parser.Parser and supports Document, Page, Chunk, and Element output modes.

go-xberg-sdk/
├── xberg/                              # Core SDK
├── components/document/parser/xberg/   # Eino Parser adapter
├── examples/                           # Core, Eino, OCR, and LLM extraction/vision examples
├── scripts/                            # Native dependency and bundle tooling
├── docs/NATIVE_BUNDLE.md               # Runtime bundle guide
└── .github/workflows/                   # CI and multi-platform releases

Distribution model

The project separates Go source distribution from native runtime delivery:

1. The Go module stays lightweight. Git tags and module zip files contain Go source, documentation, and scripts, but not large platform-specific native libraries.
2. GitHub Releases provide native bundles. Each release builds Xberg v1.0.14 from source with full-no-heic, then publishes archives and SHA-256 sidecars for the three supported OS/architecture combinations. A bundle contains libxberg_ffi, recursively discovered non-system sidecars, relative runtime paths, and licenses.

Consequently, go get does not install the native runtime automatically. Use the matching full-no-heic bundle from this project's GitHub Release when an application needs XLSX, HWP/HWPX, iWork, WordPerfect, MDX, XML, or QR support.

Requirements

• Go 1.26 or later.
• CGO_ENABLED=1 and a C compiler.
• The Go binding and native FFI must use the same Xberg version. This project currently pins both to v1.0.14.
• Building a Linux release bundle requires patchelf. macOS uses otool and install_name_tool from the system toolchain.
• Xberg v1.0.14 does not publish a macOS Intel Go native asset. Current releases therefore support macOS ARM64, Linux AMD64, and Linux ARM64.
• Linux bundles include ONNX Runtime. full-no-heic does not require libheif, but it also excludes HEIC/HEIF and Candle VLM OCR.

Installation

Install the Go packages

go get github.com/sungithubid/go-xberg-sdk/xberg

# For the Eino adapter:
go get github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg

Prepare the native FFI for development

Run Xberg's official setup command in your application module. It downloads and verifies the native library matching the Go binding and generates a local link shim for the application:

go run github.com/xberg-io/xberg/packages/go/cmd/setup@v1.0.14

This downloads the upstream default FFI and is only suitable for development that does not need its missing Excel and related features. Use this project's matching GitHub Release bundle to parse XLSX.

When developing this repository directly, use:

make native
make test

Use a GitHub Release bundle

For application distribution, download the asset matching the go-xberg-sdk version and target platform:

VERSION=v0.1.0
PLATFORM=linux-x86_64

gh release download "$VERSION" \
  --repo sungithubid/go-xberg-sdk \
  --pattern "go-xberg-sdk-native-${PLATFORM}.tar.gz*"
sha256sum -c "go-xberg-sdk-native-${PLATFORM}.tar.gz.sha256"
tar -xzf "go-xberg-sdk-native-${PLATFORM}.tar.gz"

On macOS, use shasum -a 256 -c <checksum-file>. Keep every .so or .dylib from the extracted archive in the same directory. The bundle's own README.md contains detailed deployment instructions.

Point CGO at the extracted directory when building:

CGO_ENABLED=1 \
CGO_LDFLAGS="-L/absolute/path/to/go-xberg-sdk-native-linux-x86_64" \
go build ./...

For local execution, also set LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on macOS. A distributed application can instead place all bundle libraries next to its executable and configure a $ORIGIN or @loader_path runtime search path.

Core SDK quick start

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/sungithubid/go-xberg-sdk/xberg"
)

func main() {
    extractor, err := xberg.New(nil)
    if err != nil {
        log.Fatal(err)
    }

    result, err := extractor.ExtractFile(context.Background(), "manual.docx")
    if err != nil {
        log.Fatal(err)
    }

    for _, document := range result.Results {
        fmt.Printf("MIME: %s\n%s\n", document.MimeType, document.Content)
    }
}

Extractor owns no native handle that requires cleanup, so it has no Close method. It is safe for concurrent use.

In addition to ExtractFile, the SDK supports byte slices and readers:

result, err := extractor.ExtractBytes(ctx, data,
    xberg.WithFilename("manual.docx"),
)

result, err = extractor.ExtractReader(ctx, reader,
    xberg.WithFilename("manual.pdf"),
    xberg.WithMIMEType("application/pdf"),
)

For ZIP-container formats such as DOCX, XLSX, and PPTX, always provide a filename or MIME hint when using in-memory or reader input.

Eino Parser quick start

package main

import (
    "context"
    "fmt"
    "os"

    einoparser "github.com/cloudwego/eino/components/document/parser"
    xbergparser "github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg"
)

func main() {
    ctx := context.Background()
    p, err := xbergparser.NewParser(ctx, &xbergparser.Config{
        DocumentMode: xbergparser.DocumentModeChunk,
    })
    if err != nil {
        panic(err)
    }

    file, err := os.Open("manual.pdf")
    if err != nil {
        panic(err)
    }
    defer file.Close()

    docs, err := p.Parse(ctx, file,
        einoparser.WithURI("manual.pdf"),
        einoparser.WithExtraMeta(map[string]any{"knowledge_base": "manuals"}),
    )
    if err != nil {
        panic(err)
    }

    for _, doc := range docs {
        fmt.Printf("%s: %s\n", doc.ID, doc.Content)
    }
}

See the Eino Parser Chinese guide for output modes, OCR, PDF, chunking, and metadata options. Advanced Xberg per-file settings remain accessible through WithNativeFileConfig.

Examples and verification

make test
make test-race
make vet
make example-core
make example-eino
make example-ocr OCR_INPUT=/path/to/scan.png

# 1. LLM structured metadata extraction (OpenAI-compatible API extracting title, summary, tags, entities, etc.)
export OPENAI_API_KEY="sk-..."
# export OPENAI_BASE_URL="https://api.deepseek.com/v1" # Optional OpenAI-compatible endpoint
# export OPENAI_MODEL="gpt-4o-mini"                   # Optional model
make example-llm-structured LLM_INPUT=/path/to/document.pdf

# 2. Vision/Multimodal LLM high-fidelity OCR (GPT-4o, Qwen-VL, GLM-4V, etc.)
# export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
# export OPENAI_MODEL="qwen-vl-max"
make example-llm-vision VISION_INPUT=/path/to/image-or-scanned.pdf

The OCR example deliberately has no repository-local input fixture. Supply an image or scanned PDF explicitly. LLM examples read standard OPENAI_API_KEY, OPENAI_BASE_URL, and OPENAI_MODEL environment variables, supporting OpenAI, DeepSeek, Qwen, Kimi, Ollama, and other compatible providers.

Build a native Release

Pushing a v* tag triggers the release-native workflow. On all three native runners, it:

1. Checks out Xberg v1.0.14 and builds its FFI with full-no-heic; Linux uses cargo-zigbuild to retain a glibc 2.28 floor.
2. Runs the Go integration test containing a real minimal XLSX workbook.
3. Recursively collects non-system dynamic dependencies and discoverable licenses.
4. Rewrites dependency paths to @loader_path on macOS or $ORIGIN on Linux.
5. Links and tests against the packaged directory.
6. Generates and uploads dist/*.tar.gz and matching .sha256 files.

For a manual local package, first stage a library built the same way, its header, and a FEATURES file containing full-no-heic under native/<platform>/. Then run make release-native NATIVE_PROVIDER=existing VERSION=<tag>.

Runtime boundaries

The native bundle removes implicit dependencies on build-host paths such as a Homebrew prefix, but it is not a fully static environment. It still relies on the target OS loader, base system libraries, and a compatible glibc or macOS version. Optional OCR, layout, embedding, and transcription features can also require model assets or network downloads. Test those capabilities in an offline environment matching production before deployment.

The underlying Xberg FFI call is synchronous. A context.Context deadline is converted into an Xberg timeout, and cancellation is checked before and after the native call, but cancellation cannot guarantee an immediate interruption after execution has entered native code.

License

This project is licensed under the MIT License. Licenses for Xberg, ONNX Runtime, and discoverable native sidecars are included under third_party/ and in each bundle's licenses/ directory. Review all native codec licenses before choosing a distribution model for your product.