dat

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 18, 2025 License: MIT Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// BoundaryMarker is the 8-byte marker separating fixed and dynamic data
	BoundaryMarker = "\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb"

	// NullRowSentinel is the sentinel value indicating a null row reference
	NullRowSentinel uint32 = 0xfefe_fefe

	// MinDATFileSize is the minimum size for a valid DAT file (4 bytes for row count + 8 bytes boundary)
	MinDATFileSize = 12
)

Constants for DAT file format

View Source
const CommunitySchemaURL = "https://github.com/poe-tool-dev/dat-schema/releases/download/latest/schema.min.json"

CommunitySchemaURL is the official URL for the community schema

Variables

This section is empty.

Functions

func FilterDATFiles

func FilterDATFiles(allPaths []string, languages []string, tables []string, allTables bool) []string

FilterDATFiles filters a list of file paths to find DAT files matching the specified criteria. It filters by file extension (.datc64), language, and table names based on the configuration.

Parameters:

  • allPaths: List of all file paths to filter
  • languages: List of languages to include (empty means English only)
  • tables: List of table names to include (empty when allTables is true)
  • allTables: If true, include all tables regardless of the tables parameter

Returns:

A filtered list of DAT file paths that match all criteria

func GetTableNameFromPath

func GetTableNameFromPath(path string) string

GetTableNameFromPath extracts the table name from a DAT file path. It removes the directory prefix and file extension to return just the table name.

Example:

GetTableNameFromPath("data/baseitemtypes.datc64") returns "baseitemtypes"
GetTableNameFromPath("Data/Simplified Chinese/baseitemtypes.datc64") returns "baseitemtypes"

func IsDATFile

func IsDATFile(path string) bool

IsDATFile checks if a file path represents a .datc64 DAT file. This function performs a case-insensitive check for the .datc64 extension.

Example:

IsDATFile("data/baseitemtypes.datc64") returns true
IsDATFile("data/baseitemtypes.txt") returns false
IsDATFile("data/BaseItemTypes.DATC64") returns true (case-insensitive)

func MatchesLanguage

func MatchesLanguage(path string, languages []string) bool

MatchesLanguage checks if a DAT file path matches any of the specified languages. Language-specific files have paths like "Data/Simplified Chinese/..." while default English files are directly in "Data/".

Language matching logic:

  • Files directly in "Data/" are considered English (default)
  • Files in "Data/{Language}/" are considered to be in that specific language
  • If the languages list contains "English", default files in "Data/" will match
  • If languages list is empty, no files match

Parameters:

  • path: The file path to check
  • languages: List of language names to match against

Returns:

true if the file matches any of the specified languages, false otherwise

Example:

MatchesLanguage("Data/baseitemtypes.datc64", []string{"English"}) returns true
MatchesLanguage("Data/Simplified Chinese/baseitemtypes.datc64", []string{"Simplified Chinese"}) returns true
MatchesLanguage("Data/baseitemtypes.datc64", []string{"French"}) returns false

func MatchesTable

func MatchesTable(path string, tables []string) bool

MatchesTable checks if a DAT file path matches any of the specified table names. The comparison is case-insensitive to handle variations in table name casing.

Parameters:

  • path: The file path to check
  • tables: List of table names to match against

Returns:

true if the file's table name matches any of the specified tables, false otherwise

Example:

MatchesTable("data/baseitemtypes.datc64", []string{"BaseItemTypes"}) returns true
MatchesTable("data/baseitemtypes.datc64", []string{"Skills"}) returns false
MatchesTable("data/BaseItemTypes.datc64", []string{"baseitemtypes"}) returns true (case-insensitive)

func ReadArrayMetadata

func ReadArrayMetadata(rowData []byte) (count uint64, offset uint64, err error)

ReadArrayMetadata reads array count and offset from fixed row data

Types

type ArrayReader

type ArrayReader struct {
	// contains filtered or unexported fields
}

ArrayReader provides utilities for reading arrays from DAT dynamic data

func NewArrayReader

func NewArrayReader(dynamicData []byte) *ArrayReader

NewArrayReader creates a new array reader for dynamic data

func (*ArrayReader) ReadTypedArray

func (ar *ArrayReader) ReadTypedArray(offset uint64, count uint64, elementType FieldType) (interface{}, error)

ReadTypedArray reads an array of the specified type from the given offset

type BinaryReader

type BinaryReader struct {
	// contains filtered or unexported fields
}

BinaryReader provides utilities for reading binary data from DAT files

func NewBinaryReader

func NewBinaryReader(data []byte) *BinaryReader

NewBinaryReader creates a new binary reader for the given data

func (*BinaryReader) PeekBytes

func (br *BinaryReader) PeekBytes(count int) ([]byte, error)

PeekBytes reads bytes without advancing the position

func (*BinaryReader) Position

func (br *BinaryReader) Position() int

Position returns the current read position

func (*BinaryReader) ReadBool

func (br *BinaryReader) ReadBool() (bool, error)

ReadBool reads a boolean value (single byte, 0 = false, non-zero = true)

func (*BinaryReader) ReadBytes

func (br *BinaryReader) ReadBytes(count int) ([]byte, error)

ReadBytes reads the specified number of bytes and advances the position

func (*BinaryReader) ReadFloat32

func (br *BinaryReader) ReadFloat32() (float32, error)

ReadFloat32 reads a 32-bit float in little-endian format

func (*BinaryReader) ReadFloat64

func (br *BinaryReader) ReadFloat64() (float64, error)

ReadFloat64 reads a 64-bit float in little-endian format

func (*BinaryReader) ReadInt16

func (br *BinaryReader) ReadInt16() (int16, error)

ReadInt16 reads a 16-bit signed integer in little-endian format

func (*BinaryReader) ReadInt32

func (br *BinaryReader) ReadInt32() (int32, error)

ReadInt32 reads a 32-bit signed integer in little-endian format

func (*BinaryReader) ReadInt64

func (br *BinaryReader) ReadInt64() (int64, error)

ReadInt64 reads a 64-bit signed integer in little-endian format

func (*BinaryReader) ReadNullableUint32

func (br *BinaryReader) ReadNullableUint32() (*uint32, error)

ReadNullableUint32 reads a 32-bit unsigned integer that may be null (0xfefe_fefe)

func (*BinaryReader) ReadUint8

func (br *BinaryReader) ReadUint8() (uint8, error)

ReadUint8 reads a single byte as uint8

func (*BinaryReader) ReadUint16

func (br *BinaryReader) ReadUint16() (uint16, error)

ReadUint16 reads a 16-bit unsigned integer in little-endian format

func (*BinaryReader) ReadUint32

func (br *BinaryReader) ReadUint32() (uint32, error)

ReadUint32 reads a 32-bit unsigned integer in little-endian format

func (*BinaryReader) ReadUint64

func (br *BinaryReader) ReadUint64() (uint64, error)

ReadUint64 reads a 64-bit unsigned integer in little-endian format

func (*BinaryReader) Remaining

func (br *BinaryReader) Remaining() int

Remaining returns the number of bytes remaining to read

func (*BinaryReader) Seek

func (br *BinaryReader) Seek(offset int) error

Seek sets the read position to the given offset

func (*BinaryReader) Size

func (br *BinaryReader) Size() int

Size returns the total size of the data

func (*BinaryReader) Skip

func (br *BinaryReader) Skip(bytes int) error

Skip advances the read position by the given number of bytes

type ColumnReference

type ColumnReference struct {
	Table  string  `json:"table"`            // Referenced table name
	Column *string `json:"column,omitempty"` // Referenced column name (optional)
}

ColumnReference represents a foreign key relationship from the community schema

type CommunitySchema

type CommunitySchema struct {
	SchemaMetadata
	Tables       []TableSchema       `json:"tables"`       // Table definitions
	Enumerations []SchemaEnumeration `json:"enumerations"` // Enumeration definitions
}

CommunitySchema represents the complete community schema file structure

func (*CommunitySchema) FilterTablesForVersion

func (schema *CommunitySchema) FilterTablesForVersion(tableNames []string, version string) ([]string, error)

FilterTablesForVersion filters a list of table names to only include those valid for the given version

func (*CommunitySchema) GetSchemaTableNames

func (schema *CommunitySchema) GetSchemaTableNames() []string

GetSchemaTableNames returns all table names from the schema

func (*CommunitySchema) GetTableSchema

func (cs *CommunitySchema) GetTableSchema(tableName string) (*TableSchema, bool)

GetTableSchema finds a table schema by name (case-insensitive) DEPRECATED: Use GetTableSchemaForVersion for version-aware schema selection

func (*CommunitySchema) GetTableSchemaForVersion

func (cs *CommunitySchema) GetTableSchemaForVersion(tableName string, gameVersion string) (*TableSchema, error)

GetTableSchemaForVersion finds a table schema by name filtered by game version compatibility

func (*CommunitySchema) GetValidTables

func (cs *CommunitySchema) GetValidTables(gameVersion int) []TableSchema

GetValidTables returns all tables that are valid for the given game version

type DATParser

type DATParser struct {
	// contains filtered or unexported fields
}

DATParser implements the Parser interface for Path of Exile DAT files

func (*DATParser) CalculateRowSize

func (p *DATParser) CalculateRowSize(schema *TableSchema, width ParserWidth) int

CalculateRowSize calculates the size of a single row based on the schema and parser width

func (*DATParser) ParseDATFile

func (p *DATParser) ParseDATFile(ctx context.Context, r io.Reader, schema *TableSchema) (*ParsedTable, error)

ParseDATFile parses a complete DAT file using the provided schema Uses default width detection (defaults to 64-bit for backward compatibility)

func (*DATParser) ParseDATFileWithFilename

func (p *DATParser) ParseDATFileWithFilename(ctx context.Context, r io.Reader, filename string, schema *TableSchema) (*ParsedTable, error)

ParseDATFileWithFilename parses a complete DAT file with width detection based on filename extension

func (*DATParser) ReadArray

func (p *DATParser) ReadArray(dynamicData []byte, offset uint64, count uint64, elementType FieldType) (interface{}, error)

ReadArray implements the Parser interface for reading arrays

func (*DATParser) ReadField

func (p *DATParser) ReadField(data []byte, column *TableColumn, dynamicData []byte) (interface{}, error)

ReadField implements the Parser interface for reading individual field values

func (*DATParser) ReadString

func (p *DATParser) ReadString(dynamicData []byte, offset uint64) (string, error)

ReadString implements the Parser interface for reading UTF-16 strings

type DatFile

type DatFile struct {
	// RowCount is the number of rows in the DAT file (first 4 bytes)
	RowCount int

	// RowLength is the fixed size of each row in bytes
	RowLength int

	// FixedData contains the fixed-size row data section
	FixedData []byte

	// DynamicData contains the variable-length data section (strings, arrays)
	// Starts after the boundary marker (\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb)
	DynamicData []byte
}

DatFile represents the structure of a parsed DAT file

type DataValidator

type DataValidator struct {
	// contains filtered or unexported fields
}

DataValidator provides utilities for validating DAT file data

func NewDataValidator

func NewDataValidator(options *ValidationOptions) *DataValidator

NewDataValidator creates a new data validator with the given options

func (*DataValidator) ValidateArrayCount

func (dv *DataValidator) ValidateArrayCount(count uint64) error

ValidateArrayCount validates that an array count is within reasonable limits

func (*DataValidator) ValidateArrayOffset

func (dv *DataValidator) ValidateArrayOffset(offset uint64, dataSize int) error

ValidateArrayOffset validates that an array offset is within valid bounds

func (*DataValidator) ValidateStringLength

func (dv *DataValidator) ValidateStringLength(length int) error

ValidateStringLength validates that a string length is within reasonable limits

func (*DataValidator) ValidateStringOffset

func (dv *DataValidator) ValidateStringOffset(offset uint64, dataSize int) error

ValidateStringOffset validates that a string offset is within valid bounds

type FieldBounds

type FieldBounds struct {
	Offset    int
	EndOffset int
	FieldType FieldType
	IsArray   bool
}

FieldBounds represents the discovered offset boundaries for a field

type FieldType

type FieldType string

FieldType represents a DAT file field type from the community schema

const (
	TypeBool    FieldType = "bool"
	TypeString  FieldType = "string"
	TypeInt16   FieldType = "i16"
	TypeUint16  FieldType = "u16"
	TypeInt32   FieldType = "i32"
	TypeUint32  FieldType = "u32"
	TypeInt64   FieldType = "i64"
	TypeUint64  FieldType = "u64"
	TypeFloat32 FieldType = "f32"
	TypeFloat64 FieldType = "f64"

	// Reference types
	TypeRow        FieldType = "row"        // row index (references column in same table)
	TypeForeignRow FieldType = "foreignrow" // row index (references column in foreign table)
	TypeEnumRow    FieldType = "enumrow"    // row index (references foreign table with no columns)
	TypeLongID     FieldType = "longid"     // 64-bit row reference (foreign table)

	// Array type (used when array: true and type is one of the above)
	TypeArray FieldType = "array" // column is an array of unknown type
)

FieldType constants matching the community schema specification

func (FieldType) GoType

func (ft FieldType) GoType() reflect.Type

GoType returns the corresponding Go reflect.Type for this FieldType

func (FieldType) IsArray

func (ft FieldType) IsArray(array bool) bool

IsArray returns true if this field type represents an array

func (FieldType) IsReference

func (ft FieldType) IsReference() bool

IsReference returns true if this field type represents a row reference

func (FieldType) Size

func (ft FieldType) Size() int

Size returns the fixed size in bytes for this field type in DAT files Returns 0 for variable-length types (strings, arrays)

func (FieldType) SizeForWidth

func (ft FieldType) SizeForWidth(width ParserWidth) int

SizeForWidth returns the size in bytes for this field type based on parser width Fixed to match actual .datc64 format based on poe-dat-viewer reference implementation

func (FieldType) Valid

func (ft FieldType) Valid() bool

Valid checks if the FieldType is a valid type from the community schema

type ParseMetadata

type ParseMetadata struct {
	FixedDataSize   int `json:"fixedDataSize"`   // Size of fixed data section
	DynamicDataSize int `json:"dynamicDataSize"` // Size of dynamic data section
	TotalFileSize   int `json:"totalFileSize"`   // Total DAT file size
	MaxFieldsParsed int `json:"maxFieldsParsed"` // Maximum number of fields parsed in any row (for partial schema)
}

ParseMetadata contains metadata about the parsing operation

type ParsedRow

type ParsedRow struct {
	Index        int                    `json:"index"`        // Row index (0-based)
	Fields       map[string]interface{} `json:"fields"`       // Field name to parsed value mapping
	FieldsParsed int                    `json:"fieldsParsed"` // Number of fields successfully parsed (useful for partial schema)
}

ParsedRow represents a single parsed row from a DAT table

type ParsedTable

type ParsedTable struct {
	Schema   *TableSchema   `json:"schema"`   // Original table schema
	RowCount int            `json:"rowCount"` // Number of rows parsed
	Rows     []ParsedRow    `json:"rows"`     // All parsed row data
	Metadata *ParseMetadata `json:"metadata"` // Parsing metadata
}

ParsedTable represents a completely parsed DAT table with all rows and metadata

type Parser

type Parser interface {
	// ParseDATFile parses a DAT file from the given reader using the provided schema
	ParseDATFile(ctx context.Context, r io.Reader, schema *TableSchema) (*ParsedTable, error)

	// ParseDATFileWithFilename parses a DAT file with width detection based on filename extension
	ParseDATFileWithFilename(ctx context.Context, r io.Reader, filename string, schema *TableSchema) (*ParsedTable, error)

	// ReadField reads a single field value from the DAT data
	ReadField(data []byte, column *TableColumn, dynamicData []byte) (interface{}, error)

	// ReadString reads a UTF-16 string from the dynamic data section
	ReadString(dynamicData []byte, offset uint64) (string, error)

	// ReadArray reads an array of values from the dynamic data section
	ReadArray(dynamicData []byte, offset uint64, count uint64, elementType FieldType) (interface{}, error)
}

Parser defines the interface for parsing DAT files

func NewDATParser

func NewDATParser(options *ParserOptions) Parser

This parser will stop processing fields when they would exceed the actual row size

type ParserOptions

type ParserOptions struct {
	// StrictMode enables strict validation of DAT file format
	StrictMode bool

	// ValidateReferences checks that foreign key references point to valid rows
	ValidateReferences bool

	// MaxStringLength limits the maximum length of strings to prevent memory issues
	MaxStringLength int

	// MaxArrayCount limits the maximum number of array elements
	MaxArrayCount int

	// MaxJunctionTableArrayCount limits the maximum array size for junction table creation
	// This is separate from MaxArrayCount to allow different limits for different purposes
	MaxJunctionTableArrayCount int

	// ArraySizeWarningThreshold sets the threshold for logging warnings about large arrays
	ArraySizeWarningThreshold int
}

ParserOptions configures DAT parsing behavior

func DefaultParserOptions

func DefaultParserOptions() *ParserOptions

DefaultParserOptions returns sensible default options for DAT parsing

type ParserWidth

type ParserWidth int

ParserWidth represents the bit width of the DAT file parser

const (
	// Width32 represents 32-bit DAT files (.dat, .datl)
	Width32 ParserWidth = 32
	// Width64 represents 64-bit DAT files (.dat64, .datl64)
	Width64 ParserWidth = 64
)

func WidthForFilename

func WidthForFilename(filename string) ParserWidth

WidthForFilename determines the parser width based on the DAT file extension

type SchemaEnumeration

type SchemaEnumeration struct {
	ValidFor    ValidFor  `json:"validFor"`    // Game version compatibility
	Name        string    `json:"name"`        // Enumeration name
	Indexing    int       `json:"indexing"`    // 0 or 1 based indexing
	Enumerators []*string `json:"enumerators"` // Enumeration values (nullable)
}

SchemaEnumeration represents an enumeration definition from the community schema

type SchemaManager

type SchemaManager struct {
	// contains filtered or unexported fields
}

SchemaManager implements SchemaManager using a cached schema file

func NewSchemaManager

func NewSchemaManager() (*SchemaManager, error)

LoadCachedSchema loads the schema from cache or downloads it if not available

func (*SchemaManager) GetTableSchema

func (sm *SchemaManager) GetTableSchema(tableName string) (*TableSchema, bool)

GetTableSchema retrieves a specific table schema by name DEPRECATED: Use GetTableSchemaForVersion for version-aware schema selection

func (*SchemaManager) GetTableSchemaForVersion

func (sm *SchemaManager) GetTableSchemaForVersion(tableName string, gameVersion string) (*TableSchema, error)

GetTableSchemaForVersion retrieves a table schema by name filtered by game version compatibility

func (*SchemaManager) GetValidTablesForVersion

func (sm *SchemaManager) GetValidTablesForVersion(version string) ([]TableSchema, error)

GetValidTablesForVersion returns all tables valid for the given game version

func (*SchemaManager) IsTableValidForVersion

func (sm *SchemaManager) IsTableValidForVersion(tableName, version string) (bool, error)

IsTableValidForVersion checks if a table is valid for the given game version

func (*SchemaManager) LoadSchema

func (sm *SchemaManager) LoadSchema() (*CommunitySchema, error)

LoadSchema returns the cached schema

type SchemaMetadata

type SchemaMetadata struct {
	Version   int `json:"version"`   // Schema version number
	CreatedAt int `json:"createdAt"` // Unix timestamp when schema was created
}

SchemaMetadata contains metadata about the community schema

type StringReader

type StringReader struct {
	// contains filtered or unexported fields
}

StringReader provides utilities for reading strings from DAT dynamic data

func NewStringReader

func NewStringReader(dynamicData []byte) *StringReader

NewStringReader creates a new string reader for dynamic data

func (*StringReader) ReadUTF16String

func (sr *StringReader) ReadUTF16String(offset uint64) (string, error)

ReadUTF16String reads a null-terminated UTF-16 string from the given offset

func (*StringReader) ReadUTF32String

func (sr *StringReader) ReadUTF32String(offset uint64) (string, error)

ReadUTF32String reads a null-terminated UTF-32 string from the given offset

type TableColumn

type TableColumn struct {
	Name        *string          `json:"name"`
	Description *string          `json:"description"`
	Array       bool             `json:"array"`
	Type        FieldType        `json:"type"`
	Unique      bool             `json:"unique"`
	Localized   bool             `json:"localized"`
	Until       *string          `json:"until"`      // Version when this column was removed
	References  *ColumnReference `json:"references"` // Foreign key reference
	File        *string          `json:"file"`       // File extension for asset files
	Files       []string         `json:"files"`      // Multiple file extensions
	Interval    bool             `json:"interval"`   // Whether this is an interval field
}

TableColumn represents a column definition from the community schema

type TableSchema

type TableSchema struct {
	ValidFor ValidFor      `json:"validFor"` // Game version compatibility
	Name     string        `json:"name"`     // Table name (matches DAT filename)
	Columns  []TableColumn `json:"columns"`  // Column definitions
	Tags     []string      `json:"tags"`     // Metadata tags
}

TableSchema represents a complete table definition from the community schema

func (*TableSchema) GetArrayColumns

func (table *TableSchema) GetArrayColumns() []TableColumn

GetArrayColumns returns all columns in a table that are arrays

func (*TableSchema) GetColumnByName

func (table *TableSchema) GetColumnByName(columnName string) (*TableColumn, bool)

GetColumnByName finds a column by name in a table schema

func (*TableSchema) GetColumnNames

func (table *TableSchema) GetColumnNames() []string

GetColumnNames returns all column names for a given table

func (*TableSchema) GetForeignKeyColumns

func (table *TableSchema) GetForeignKeyColumns() []TableColumn

GetForeignKeyColumns returns all columns in a table that are foreign key references

func (*TableSchema) GetLocalizedColumns

func (table *TableSchema) GetLocalizedColumns() []TableColumn

GetLocalizedColumns returns all columns in a table that are localized

func (*TableSchema) HasColumn

func (table *TableSchema) HasColumn(columnName string) bool

HasColumn checks if a table has a column with the given name

type ValidFor

type ValidFor int

ValidFor represents game version compatibility flags

const (
	ValidForPoE1 ValidFor = 0x01 // Path of Exile 1 only
	ValidForPoE2 ValidFor = 0x02 // Path of Exile 2 only
	ValidForBoth ValidFor = 0x03 // Both games (PoE1 | PoE2)
)

func (ValidFor) IsValidForGame

func (vf ValidFor) IsValidForGame(gameVersion int) bool

IsValidForGame checks if a ValidFor flag is compatible with the given game version

type ValidationOptions

type ValidationOptions struct {
	// MaxStringLength is the maximum allowed string length
	MaxStringLength int

	// MaxArrayCount is the maximum allowed array element count
	MaxArrayCount int

	// ValidateStringOffsets checks that string offsets are within bounds
	ValidateStringOffsets bool

	// ValidateArrayOffsets checks that array offsets are within bounds
	ValidateArrayOffsets bool
}

ValidationOptions configures data validation behavior

func DefaultValidationOptions

func DefaultValidationOptions() *ValidationOptions

DefaultValidationOptions returns sensible default validation options

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL