base

Package base — shared helpers for database adapters

Overview

base holds the reusable pieces every database adapter needs, so the SQLite, PostgreSQL, MS SQL Server and MySQL adapters do not each carry their own copy.

The problem

Before base existed:

• about 800 lines of duplicated export code across four adapters
• about 600 lines of duplicated import code across four adapters
• about 300 lines of duplicated type conversion
• 1700 lines of duplication in total — a third of the adapters' code

The solution

base centralises the shared logic into reusable components:

pkg/adapters/base/
├── export_helper.go      - shared export logic
├── import_helper.go      - shared import logic
├── type_converter.go     - type conversion for every adapter
├── sql_adapter.go        - SQL dialect differences
└── doc.go                - package documentation

The components

1. ExportHelper

Shared logic for exporting data into TDTP packets.

Methods:

• ExportTable() — export a whole table
• ExportTableWithQuery() — export with TDTQL filtering, pushed down into SQL
• ExportTableIncremental() — incremental synchronisation

Interfaces:

type SchemaReader interface {
    GetTableSchema(ctx context.Context, tableName string) (packet.Schema, error)
}

type DataReader interface {
    ReadAllRows(ctx context.Context, tableName string, schema packet.Schema) ([][]string, error)
    ReadRowsWithSQL(ctx context.Context, sql string, schema packet.Schema) ([][]string, error)
    GetRowCount(ctx context.Context, tableName string) (int64, error)
}

type SQLAdapter interface {
    AdaptSQL(standardSQL string, tableName string, schema packet.Schema, query *packet.Query) string
}

2. ImportHelper

Shared logic for importing TDTP packets into a database.

Methods:

• ImportPacket() — import one packet
• ImportPackets() — import several packets atomically
• temporary tables, for an atomic replacement

Interfaces:

type TableManager interface {
    TableExists(ctx context.Context, tableName string) (bool, error)
    CreateTable(ctx context.Context, tableName string, schema packet.Schema) error
    DropTable(ctx context.Context, tableName string) error
    RenameTable(ctx context.Context, oldName, newName string) error
}

type DataInserter interface {
    InsertRows(ctx context.Context, tableName string, schema packet.Schema,
               rows []packet.Row, strategy adapters.ImportStrategy) error
}

3. UniversalTypeConverter

Type conversion between a database's types and TDTP's.

Methods:

• ConvertValueToTDTP() — database value to TDTP
• DBValueToString() — database value to a string, respecting that database's quirks
• TypedValueToSQL() — TDTP value to a SQL parameter for a prepared statement

Database-specific types handled:

• PostgreSQL: UUID, JSONB, INET, ARRAY, NUMERIC
• MS SQL Server: UNIQUEIDENTIFIER, TIMESTAMP/ROWVERSION, NVARCHAR

4. SQLAdapter

Papers over SQL dialect differences.

Implementations:

• StandardSQLAdapter — SQLite, PostgreSQL, MySQL (LIMIT/OFFSET)
• MSSQLAdapter — MS SQL Server (OFFSET/FETCH)

Usage

Step 1: implement the interfaces in your adapter

package sqlite

import (
    "github.com/ruslano69/tdtp-framework/pkg/adapters/base"
)

type Adapter struct {
    db           *sql.DB
    exportHelper *base.ExportHelper
    importHelper *base.ImportHelper
    converter    *base.UniversalTypeConverter
}

// SchemaReader
func (a *Adapter) GetTableSchema(ctx context.Context, tableName string) (packet.Schema, error) {
    // The SQLite-specific part
}

// DataReader
func (a *Adapter) ReadAllRows(ctx context.Context, tableName string, schema packet.Schema) ([][]string, error) {
    // The SQLite-specific part
}

func (a *Adapter) ReadRowsWithSQL(ctx context.Context, sql string, schema packet.Schema) ([][]string, error) {
    // The SQLite-specific part
}

func (a *Adapter) GetRowCount(ctx context.Context, tableName string) (int64, error) {
    // The SQLite-specific part
}

// TableManager
func (a *Adapter) TableExists(ctx context.Context, tableName string) (bool, error) { ... }
func (a *Adapter) CreateTable(ctx context.Context, tableName string, schema packet.Schema) error { ... }
func (a *Adapter) DropTable(ctx context.Context, tableName string) error { ... }
func (a *Adapter) RenameTable(ctx context.Context, oldName, newName string) error { ... }

// DataInserter
func (a *Adapter) InsertRows(ctx context.Context, tableName string, schema packet.Schema,
                             rows []packet.Row, strategy adapters.ImportStrategy) error { ... }

Step 2: wire up the helpers

func NewAdapter(dsn string) (*Adapter, error) {
    db, err := sql.Open("sqlite", dsn)
    if err != nil {
        return nil, err
    }

    a := &Adapter{db: db}

    // The converter
    a.converter = base.NewUniversalTypeConverter()

    // The export helper — SQLite needs no SQL adapter
    a.exportHelper = base.NewExportHelper(a, a, a.converter, nil)

    // The import helper, with temporary tables
    a.importHelper = base.NewImportHelper(a, a, a, true)

    return a, nil
}

Step 3: delegate the Adapter interface methods

// Delegate the export
func (a *Adapter) ExportTable(ctx context.Context, tableName string) ([]*packet.DataPacket, error) {
    return a.exportHelper.ExportTable(ctx, tableName)
}

func (a *Adapter) ExportTableWithQuery(ctx context.Context, tableName string,
                                       query *packet.Query, sender, recipient string) ([]*packet.DataPacket, error) {
    return a.exportHelper.ExportTableWithQuery(ctx, tableName, query, sender, recipient)
}

// Delegate the import
func (a *Adapter) ImportPacket(ctx context.Context, pkt *packet.DataPacket,
                               strategy adapters.ImportStrategy) error {
    return a.importHelper.ImportPacket(ctx, pkt, strategy)
}

func (a *Adapter) ImportPackets(ctx context.Context, packets []*packet.DataPacket,
                                strategy adapters.ImportStrategy) error {
    return a.importHelper.ImportPackets(ctx, packets, strategy)
}

Examples per database

SQLite

a.exportHelper = base.NewExportHelper(a, a, a.converter, nil) // nil = no SQL adaptation needed
a.importHelper = base.NewImportHelper(a, a, a, true)          // true = use temp tables

PostgreSQL

sqlAdapter := base.NewStandardSQLAdapter("postgres", a.schema+".", "\"")
a.exportHelper = base.NewExportHelper(a, a, a.converter, sqlAdapter)
a.importHelper = base.NewImportHelper(a, a, a, true)

MS SQL Server

sqlAdapter := base.NewMSSQLAdapter(a.schema) // "dbo" by default
a.exportHelper = base.NewExportHelper(a, a, a.converter, sqlAdapter)
a.importHelper = base.NewImportHelper(a, a, a, true)

MySQL

sqlAdapter := base.NewStandardSQLAdapter("mysql", "", "`")
a.exportHelper = base.NewExportHelper(a, a, a.converter, sqlAdapter)
a.importHelper = base.NewImportHelper(a, a, a, true)

What it bought

Why it is worth it

• No duplication — the shared logic lives in one place
• Easier maintenance — one change reaches every adapter
• Consistency — every adapter behaves the same way
• A new adapter is quick — you write only what is specific to it
• Testable — the helpers can be tested on their own
• Compatible — works with ETL, streaming and compression

Compatibility

It works with:

• pkg/core/packet — generating and parsing TDTP packets
• pkg/core/schema — the type system
• pkg/core/tdtql — the query language and its SQL pushdown
• pkg/etl — ETL pipelines
• every existing adapter

Testing

Write unit tests against the helpers:

func TestExportHelper_ExportTable(t *testing.T) {
    // Mock SchemaReader, DataReader
    // Test ExportTable()
}

func TestImportHelper_ImportPacket(t *testing.T) {
    // Mock TableManager, DataInserter
    // Test ImportPacket()
}

func TestUniversalTypeConverter_ConvertValueToTDTP(t *testing.T) {
    // Test conversion across the types
}

Migrating an existing adapter

See MIGRATION_EXAMPLE.md for the SQLite adapter's migration, worked through step by step.

Where to look next

If something here is unclear:

• pkg/adapters/base/doc.go — the full package documentation
• MIGRATION_EXAMPLE.md — a complete before-and-after
• docs/DEVELOPER_GUIDE.md — the framework's architecture, and how to write an adapter

Version: 1.0 Created: 2025-12-25 Author: Claude Code, as part of the refactoring initiative