go-xberg-sdk

go-xberg-sdk

English | 简体中文

A lightweight Go document extraction SDK built on the Xberg Go binding v1.0.14, with a CloudWeGo Eino Parser adapter. It converts PDFs, Office documents, images, email, web content, and structured text into Markdown, pages, chunks, or semantic elements suitable for RAG pipelines.

Packages

• github.com/sungithubid/go-xberg-sdk/xberg is the framework-independent core SDK. It accepts file paths, []byte, and io.Reader inputs.
• github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg implements Eino's parser.Parser and supports Document, Page, Chunk, and Element output modes.

go-xberg-sdk/
├── xberg/                              # Core SDK
├── components/document/parser/xberg/   # Eino Parser adapter
├── examples/                           # Core, Eino, and OCR examples
├── scripts/                            # Native dependency and bundle tooling
├── docs/NATIVE_BUNDLE.md               # Runtime bundle guide
└── .github/workflows/                   # CI and multi-platform releases

Distribution model

The project separates Go source distribution from native runtime delivery:

1. The Go module stays lightweight. Git tags and module zip files contain Go source, documentation, and scripts, but not large platform-specific native libraries.
2. GitHub Releases provide native bundles. Each release publishes archives and SHA-256 sidecars for the three supported OS/architecture combinations. A bundle contains libxberg_ffi, recursively discovered non-system sidecars, relative runtime paths, and licenses.

Consequently, go get does not install the native runtime automatically. Use Xberg's official setup command for development and the matching complete bundle from this project's GitHub Release when distributing an application.

Requirements

• Go 1.26 or later.
• CGO_ENABLED=1 and a C compiler.
• The Go binding and native FFI must use the same Xberg version. This project currently pins both to v1.0.14.
• Building a Linux release bundle requires patchelf. macOS uses otool and install_name_tool from the system toolchain.
• Xberg v1.0.14 does not publish a macOS Intel Go native asset. Current releases therefore support macOS ARM64, Linux AMD64, and Linux ARM64.
• Xberg's Linux Go assets include ONNX Runtime. A macOS build host must have libheif installed.

Installation

Install the Go packages

go get github.com/sungithubid/go-xberg-sdk/xberg

# For the Eino adapter:
go get github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg

Prepare the native FFI for development

Run Xberg's official setup command in your application module. It downloads and verifies the native library matching the Go binding and generates a local link shim for the application:

go run github.com/xberg-io/xberg/packages/go/cmd/setup@v1.0.14

When developing this repository directly, use:

make native
make test

Use a GitHub Release bundle

For application distribution, download the asset matching the go-xberg-sdk version and target platform:

VERSION=v0.1.0
PLATFORM=linux-x86_64

gh release download "$VERSION" \
  --repo sungithubid/go-xberg-sdk \
  --pattern "go-xberg-sdk-native-${PLATFORM}.tar.gz*"
sha256sum -c "go-xberg-sdk-native-${PLATFORM}.tar.gz.sha256"
tar -xzf "go-xberg-sdk-native-${PLATFORM}.tar.gz"

On macOS, use shasum -a 256 -c <checksum-file>. Keep every .so or .dylib from the extracted archive in the same directory. The bundle's own README.md contains detailed deployment instructions.

Point CGO at the extracted directory when building:

CGO_ENABLED=1 \
CGO_LDFLAGS="-L/absolute/path/to/go-xberg-sdk-native-linux-x86_64" \
go build ./...

For local execution, also set LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on macOS. A distributed application can instead place all bundle libraries next to its executable and configure a $ORIGIN or @loader_path runtime search path.

Core SDK quick start

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/sungithubid/go-xberg-sdk/xberg"
)

func main() {
    extractor, err := xberg.New(nil)
    if err != nil {
        log.Fatal(err)
    }

    result, err := extractor.ExtractFile(context.Background(), "manual.docx")
    if err != nil {
        log.Fatal(err)
    }

    for _, document := range result.Results {
        fmt.Printf("MIME: %s\n%s\n", document.MimeType, document.Content)
    }
}

Extractor owns no native handle that requires cleanup, so it has no Close method. It is safe for concurrent use.

In addition to ExtractFile, the SDK supports byte slices and readers:

result, err := extractor.ExtractBytes(ctx, data,
    xberg.WithFilename("manual.docx"),
)

result, err = extractor.ExtractReader(ctx, reader,
    xberg.WithFilename("manual.pdf"),
    xberg.WithMIMEType("application/pdf"),
)

For ZIP-container formats such as DOCX, XLSX, and PPTX, always provide a filename or MIME hint when using in-memory or reader input.

Eino Parser quick start

package main

import (
    "context"
    "fmt"
    "os"

    einoparser "github.com/cloudwego/eino/components/document/parser"
    xbergparser "github.com/sungithubid/go-xberg-sdk/components/document/parser/xberg"
)

func main() {
    ctx := context.Background()
    p, err := xbergparser.NewParser(ctx, &xbergparser.Config{
        DocumentMode: xbergparser.DocumentModeChunk,
    })
    if err != nil {
        panic(err)
    }

    file, err := os.Open("manual.pdf")
    if err != nil {
        panic(err)
    }
    defer file.Close()

    docs, err := p.Parse(ctx, file,
        einoparser.WithURI("manual.pdf"),
        einoparser.WithExtraMeta(map[string]any{"knowledge_base": "manuals"}),
    )
    if err != nil {
        panic(err)
    }

    for _, doc := range docs {
        fmt.Printf("%s: %s\n", doc.ID, doc.Content)
    }
}

See the Eino Parser Chinese guide for output modes, OCR, PDF, chunking, and metadata options. Advanced Xberg per-file settings remain accessible through WithNativeFileConfig.

Examples and verification

make test
make test-race
make vet
make example-core
make example-eino
make example-ocr OCR_INPUT=/path/to/scan.png

The OCR example deliberately has no repository-local input fixture. Supply an image or scanned PDF explicitly.

Build a native Release

Run the following command on a native runner for the target platform:

make release-native VERSION=v0.1.0

It performs the following operations:

1. Downloads and verifies the Xberg v1.0.14 FFI from the official Release.
2. Runs the Go tests.
3. Recursively collects non-system dynamic dependencies and discoverable licenses.
4. Rewrites dependency paths to @loader_path on macOS or $ORIGIN on Linux.
5. Links and tests against the packaged directory.
6. Generates dist/*.tar.gz and matching .sha256 files.

Pushing a v* tag triggers the release-native workflow, which runs the same process on all three supported native runners and uploads the assets to a GitHub Release.

Runtime boundaries

The native bundle removes implicit dependencies on build-host paths such as a Homebrew prefix, but it is not a fully static environment. It still relies on the target OS loader, base system libraries, and a compatible glibc or macOS version. Optional OCR, layout, embedding, and transcription features can also require model assets or network downloads. Test those capabilities in an offline environment matching production before deployment.

The underlying Xberg FFI call is synchronous. A context.Context deadline is converted into an Xberg timeout, and cancellation is checked before and after the native call, but cancellation cannot guarantee an immediate interruption after execution has entered native code.

License

This project is licensed under the MIT License. Licenses for Xberg, ONNX Runtime, and discoverable native sidecars are included under third_party/ and in each bundle's licenses/ directory. Review all native codec licenses before choosing a distribution model for your product.