storobj

package
v1.38.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 16, 2026 License: BSD-3-Clause Imports: 22 Imported by: 2

Documentation

Index

Constants

View Source 
const MarshallerV1HeaderLen = marshallerV1HeaderLen

MarshallerV1HeaderLen bounds the value prefix a digest-only scan must retain: DocIDAndTimeFromBinary reads no further than this.

Variables

This section is empty.

Functions

func AddOwnership added in v1.19.0 

func AddOwnership(objs []*Object, node, shard string)

func DocIDAndTimeFromBinary added in v1.26.0 

func DocIDAndTimeFromBinary(in []byte) (uint64, int64, error)

func DocIDFromBinary 

func DocIDFromBinary(in []byte) (uint64, error)

func MultiVectorFromBinary added in v1.29.0 

func MultiVectorFromBinary(in []byte, buffer []float32, targetVector string) ([][]float32, error)

func NewErrNotFoundf 

func NewErrNotFoundf(docID uint64, msg string, args ...interface{}) error

func ParseAndExtractBoolArrayProp added in v1.21.2 

func ParseAndExtractBoolArrayProp(data []byte, propName string) ([]bool, bool, error)

func ParseAndExtractNumberArrayProp 

func ParseAndExtractNumberArrayProp(data []byte, propName string) ([]float64, bool, error)

func ParseAndExtractProperty 

func ParseAndExtractProperty(data []byte, propName string) ([]string, bool, error)

func ParseAndExtractTextProp 

func ParseAndExtractTextProp(data []byte, propName string) ([]string, bool, error)

func PatchDocID added in v1.37.11 

func PatchDocID(in []byte, docID uint64) error

PatchDocID overwrites the docID in a marshalled (version 1) object binary in place, so a target can store propagated raw bytes under its own docID.

func SearchResults 

func SearchResults(in []*Object, additional additional.Properties, tenant string) search.Results

func SearchResultsWithDists 

func SearchResultsWithDists(in []*Object, addl additional.Properties,
	dists []float32,
) search.Results

func SearchResultsWithScore added in v1.24.5 

func SearchResultsWithScore(in []*Object, scores []float32, additional additional.Properties, tenant string) search.Results

func UnmarshalProperties added in v1.27.0 

func UnmarshalProperties(data []byte, properties map[string]interface{}, propertyPaths [][]string) error

UnmarshalProperties accepts serialized properties as data and populates resultProperties map with the properties specified by propertyPaths.

func UnmarshalPropertiesFromObject 

func UnmarshalPropertiesFromObject(data []byte, resultProperties map[string]interface{}, propertyPaths [][]string) error

UnmarshalPropertiesFromObject accepts marshaled object as data and populates resultProperties map with the properties specified by propertyPaths.

Check MarshalBinary for the order of elements in the input array

func VectorFromBinary 

func VectorFromBinary(in []byte, buffer []float32, targetVector string) ([]float32, error)

Types

type ErrNotFound 

type ErrNotFound struct {
	DocID       uint64
	OriginalMsg string
}

func (ErrNotFound) Error 

func (err ErrNotFound) Error() string

type ErrTargetVectorNotFound added in v1.34.20 

type ErrTargetVectorNotFound struct {
	TargetVector string
}

func (ErrTargetVectorNotFound) Error added in v1.34.20 

func (err ErrTargetVectorNotFound) Error() string

type ExportFields added in v1.37.0 

type ExportFields struct {
	ID           string
	CreateTime   int64
	UpdateTime   int64
	VectorBytes  []byte // raw LE float32 bytes, nil if no vector
	Properties   []byte // raw JSON bytes as originally stored, nil if no properties
	NamedVectors []byte // JSON-encoded named vectors, nil if none
	MultiVectors []byte // JSON-encoded multi-vectors, nil if none
}

ExportFields contains export-ready data extracted directly from the binary storage format. This avoids the full *Object deserialization round-trip (json.Unmarshal + enrichSchemaTypes + json.Marshal) that is unnecessary when the goal is to produce Parquet export rows.

func ExportFieldsFromBinary added in v1.37.0 

func ExportFieldsFromBinary(data []byte) (ExportFields, error)

ExportFieldsFromBinary extracts export-ready fields directly from the binary storage format without constructing a full *Object. This skips:

  • json.Unmarshal / json.Marshal of properties (biggest win)
  • enrichSchemaTypes (lossy type conversions not needed for export)
  • []float32 allocation for primary/named/multi vectors
  • models.Object construction overhead

type Object 

type Object struct {
	MarshallerVersion uint8
	Object            models.Object `json:"object"`
	Vector            []float32     `json:"vector"`
	VectorLen         int           `json:"-"`
	BelongsToNode     string        `json:"-"`
	BelongsToShard    string        `json:"-"`
	IsConsistent      bool          `json:"-"`
	DocID             uint64
	Vectors           map[string][]float32   `json:"vectors"`
	MultiVectors      map[string][][]float32 `json:"multivectors"`

	// PrecomputedDiskBinary, when set, is persisted verbatim (after a docID patch)
	// instead of re-marshalling. Set on the raw-propagation write path.
	PrecomputedDiskBinary []byte `json:"-"`
}

func FromBinaryDisk added in v1.37.3 

func FromBinaryDisk(data []byte, className string) (*Object, error)

FromBinaryDisk decodes a payload and stamps the caller-supplied class name on the decoded object, ignoring whatever is in the on-disk class-name field. className must be non-empty; the function returns an error otherwise. Use FromBinaryNetwork for the on-disk-fallback path used by wire-receive callers that have no canonical class to supply.

func FromBinaryDiskWithProps added in v1.37.10 

func FromBinaryDiskWithProps(data []byte, className string, properties *PropertyExtraction) (*Object, error)

FromBinaryDiskWithProps is FromBinaryDisk with a known property set; see UnmarshalBinaryDiskWithProps for the contract.

func FromBinaryNetwork added in v1.37.3 

func FromBinaryNetwork(data []byte) (*Object, error)

FromBinaryNetwork decodes a payload and reads Object.Class from the data bytes. Network methods are used for node to node communication. Use FromBinaryDisk when reading from disk, as the on-disk class-name may be empty.

func FromBinaryOptionalDisk added in v1.37.3 

func FromBinaryOptionalDisk(data []byte, className string,
	addProp additional.Properties, properties *PropertyExtraction,
) (*Object, error)

FromBinaryOptionalDisk lets the caller supply an authoritative class name; an empty className falls back to the on-disk bytes.

func FromBinaryOptionalNetwork added in v1.37.3 

func FromBinaryOptionalNetwork(data []byte,
	addProp additional.Properties, properties *PropertyExtraction,
) (*Object, error)

FromBinaryOptionalNetwork decodes a payload, optionally including/excluding vectors and properties via addProp, and reads Object.Class from the on-disk bytes. The on-disk class is not authoritative; use FromBinaryOptionalDisk when you have a className in scope.

func FromBinaryUUIDOnlyDisk added in v1.37.3 

func FromBinaryUUIDOnlyDisk(data []byte, className string) (*Object, error)

FromBinaryUUIDOnlyDisk is a header-only fast path that decodes the ID, doc ID, timestamps, and class — vectors, properties, and the rest of the payload are skipped. The caller-supplied class name is stamped on the decoded object, ignoring the on-disk class-name field. className must be non-empty; the function returns an error otherwise. There is no FromBinaryUUIDOnlyNetwork variant — this fast path runs only against bucket reads where a bucket className is always available.

func FromObject 

func FromObject(object *models.Object, vector []float32, vectors map[string][]float32, multivectors map[string][][]float32) *Object

TODO: temporary solution

func New 

func New(docID uint64) *Object

func ObjectsByDocID 

func ObjectsByDocID(bucket bucket, ids []uint64,
	additional additional.Properties, properties []string, logger logrus.FieldLogger,
) ([]*Object, error)

ObjectsByDocID resolves a batch of doc IDs against the given bucket and returns the decoded objects, dropping entries whose payload is missing. Object.Class on each decoded object is stamped from bucket.ClassName(); the bucket must have been opened with lsmkv.WithClassName, otherwise bucket.ClassName() returns an error and the call fails.

func ObjectsByDocIDWithEmpty added in v1.34.16 

func ObjectsByDocIDWithEmpty(bucket bucket, ids []uint64,
	additional additional.Properties, properties []string, logger logrus.FieldLogger,
) ([]*Object, error)

ObjectsByDocIDWithEmpty is like ObjectsByDocID but preserves nil entries at positions where a doc ID has no payload, so the returned slice always has the same length as ids. Object.Class is stamped from bucket.ClassName() — see ObjectsByDocID for the bucket-resolution contract.

func (*Object) AdditionalProperties 

func (ko *Object) AdditionalProperties() models.AdditionalProperties

AdditionalProperties groups all properties which are stored with the object and not generated at runtime

func (*Object) Class 

func (ko *Object) Class() schema.ClassName

func (*Object) CreationTimeUnix 

func (ko *Object) CreationTimeUnix() int64

func (*Object) DeepCopyDangerous 

func (ko *Object) DeepCopyDangerous() *Object

DeepCopyDangerous creates a deep copy of the underlying Object WARNING: This was purpose built for the batch ref usecase and only covers the situations that are required there. This means that cases which aren't reflected in that usecase may still contain references. Thus the suffix "Dangerous". If needed, make sure everything is copied and remove the suffix.

func (*Object) ExplainScore 

func (ko *Object) ExplainScore() string

func (*Object) GetDocID added in v1.24.0 

func (ko *Object) GetDocID() uint64

func (*Object) GetVectors added in v1.27.21 

func (ko *Object) GetVectors() models.Vectors

func (*Object) ID 

func (ko *Object) ID() strfmt.UUID

func (*Object) IterateThroughVectorDimensions added in v1.30.0 

func (ko *Object) IterateThroughVectorDimensions(f func(targetVector string, dims int) error) error

IterateThroughVectorDimensions iterates through all vectors present on the Object and invokes the callback with target name and dimensions of the vector.

func (*Object) LastUpdateTimeUnix 

func (ko *Object) LastUpdateTimeUnix() int64

func (*Object) MarshalBinary 

func (ko *Object) MarshalBinary() ([]byte, error)

func (*Object) MarshalBinaryDisk added in v1.38.0 

func (ko *Object) MarshalBinaryDisk(skipClassName bool) ([]byte, error)

MarshalBinaryDisk produces the on-disk representation of the object. When skipClassName is true, the className body is omitted (the 2-byte length prefix is written as 0). Disk readers stamp the className from the bucket onto the decoded object via FromBinaryDisk.

func (*Object) MarshalBinaryOptional added in v1.34.11 

func (ko *Object) MarshalBinaryOptional(addProps additional.Properties) ([]byte, error)

MarshalBinaryOptional creates the binary representation of a kind object, but conditionally includes vectors and properties based on the additional.Properties parameter. This is useful for network transmission where vectors may not be needed.

If addProps.Vector is false, the main vector will be serialized with length 0. If addProps.NoProps is true, the properties will be serialized with length 0. For target vectors (Vectors and MultiVectors):

  • If addProps.IncludeAllTargetVectors is true, ALL target vectors are included
  • If addProps.IncludeAllTargetVectors is false and addProps.Vectors is empty, NO target vectors are included
  • If addProps.IncludeAllTargetVectors is false and addProps.Vectors has specific names, only those are included

func (*Object) PrepareMarshalOptional added in v1.38.5 

func (ko *Object) PrepareMarshalOptional(addProps additional.Properties) (PreparedMarshal, error)

PrepareMarshalOptional runs the sizing pass of the two-pass marshal with the same addProps filtering as MarshalBinaryOptional; MarshalTo produces identical bytes.

func (*Object) Properties 

func (ko *Object) Properties() models.PropertySchema

func (*Object) PropertiesWithAdditional 

func (ko *Object) PropertiesWithAdditional(
	additional additional.Properties,
) models.PropertySchema

func (*Object) SearchResult 

func (ko *Object) SearchResult(additional additional.Properties, tenant string) *search.Result

func (*Object) SearchResultWithDist 

func (ko *Object) SearchResultWithDist(addl additional.Properties, dist float32) search.Result

func (*Object) SearchResultWithScore added in v1.24.0 

func (ko *Object) SearchResultWithScore(addl additional.Properties, score float32) search.Result

func (*Object) SearchResultWithScoreAndTenant added in v1.24.5 

func (ko *Object) SearchResultWithScoreAndTenant(addl additional.Properties, score float32, tenant string) search.Result

func (*Object) SetClass 

func (ko *Object) SetClass(class string)

func (*Object) SetDocID 

func (ko *Object) SetDocID(id uint64)

func (*Object) SetID 

func (ko *Object) SetID(id strfmt.UUID)

func (*Object) SetProperties 

func (ko *Object) SetProperties(schema models.PropertySchema)

func (*Object) UnmarshalBinaryDisk added in v1.37.3 

func (ko *Object) UnmarshalBinaryDisk(data []byte, className string) error

UnmarshalBinaryDisk decodes onto ko and stamps the supplied className on Object.Class, skipping the on-disk class-name bytes. className must be non-empty.

func (*Object) UnmarshalBinaryDiskWithProps added in v1.37.10 

func (ko *Object) UnmarshalBinaryDiskWithProps(data []byte, className string, properties *PropertyExtraction) error

UnmarshalBinaryDiskWithProps is UnmarshalBinaryDisk with a known property set: a non-nil properties decodes the property blob via UnmarshalProperties (no reflection) instead of json.Unmarshal. Names not listed are dropped, so callers must pass every property that can be present on disk.

func (*Object) UnmarshalBinaryNetwork added in v1.37.3 

func (ko *Object) UnmarshalBinaryNetwork(data []byte) error

UnmarshalBinaryNetwork is the object-method form of FromBinaryNetwork: it decodes onto ko and reads Object.Class from the data bytes. Network methods are used for node to node communication. Use UnmarshalBinaryDisk when reading from disk, as the on-disk class-name may be empty.

func (*Object) Valid 

func (ko *Object) Valid() bool

func (*Object) VectorWeights 

func (ko *Object) VectorWeights() models.VectorWeights

type PreparedMarshal added in v1.38.5 

type PreparedMarshal struct {
	// contains filtered or unexported fields
}

PreparedMarshal is the sizing-pass output of the two-pass binary marshal (see PrepareMarshalOptional, MarshalTo), letting callers that batch many objects into one payload allocate a single exactly-sized buffer instead of one per object.

It aliases rather than copies the source Object's vector slices, so concurrent mutation of vector contents is as unsafe as for MarshalBinary; otherwise it is self-contained once the source Object is discarded.

func (*PreparedMarshal) Len added in v1.38.5 

func (pm *PreparedMarshal) Len() int

Len returns the exact number of bytes MarshalTo will write.

func (*PreparedMarshal) MarshalTo added in v1.38.5 

func (pm *PreparedMarshal) MarshalTo(buf []byte) error

MarshalTo writes the serialized object into buf, which must be exactly Len() bytes long, producing the same bytes as MarshalBinaryOptional for the addProps the PreparedMarshal was prepared with.

type PropertyExtraction added in v1.27.0 

type PropertyExtraction struct {
	PropertyPaths [][]string
}

func AllPropertiesExtraction added in v1.37.10 

func AllPropertiesExtraction(class *models.Class) *PropertyExtraction

AllPropertiesExtraction lists every top-level property of the class for the jsonparser-based decode path. Returns nil (json.Unmarshal fallback) when the class is unknown or has no properties.

func NewPropExtraction added in v1.30.0 

func NewPropExtraction() *PropertyExtraction

func (*PropertyExtraction) Add added in v1.30.0 

func (pe *PropertyExtraction) Add(props ...string) *PropertyExtraction

type Vectors added in v1.24.0 

type Vectors map[string][]float32

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.