store

package

v0.6.5 Latest Latest Go to latest Published: Jan 24, 2024 License: GPL-3.0 Imports: 6 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/efixler/scrape

Links

Open Source Insights

Documentation ¶

Overview ¶

Key generation and related methods relevant to any storage backend.

Index ¶

Constants
Variables
func Key(url URLWithHostname) uint64
type DatabaseOptions
type Factory
type Maintainable
type Observable
type URLDataStore
type URLString
- func (u URLString) Hostname() string
- func (u URLString) String() string
type URLWithHostname

Constants ¶

View Source

const (
	MASK_56       uint64 = 0xffffffffffffff
	CHECKSUM_MASK uint64 = 127 << 56
)

Variables ¶

View Source

var (
	ErrorDatabaseNotFound = errors.New("database not found")
	ErrorResourceNotFound = errors.New("resource not found in data store")
)

Functions ¶

func Key ¶

func Key(url URLWithHostname) uint64

Produces a 63 bit uint contained in a uint64 (SQLite cannot accept uint64 with high bit set as a primary key) [Bit 63] Always 0 [Bits 62-56] A 7 bit checksum based on the domain name [Bits 55-0] A 56 bit hash of the URL (reduced from a 64 bit fnv1a hash)

Types ¶

type DatabaseOptions ¶

type DatabaseOptions fmt.Stringer

type Factory ¶

type Factory func() (URLDataStore, error)

type Maintainable ¶

type Maintainable interface {
	Create() error
	Clear() error
	Maintain() error
}

This interface adds create/clear/maintain methods to the URLDataStore interface.

type Observable ¶ added in v0.6.5

type Observable interface {
	Stats() (any, error)
}

This interface is to expose a method to supply data to healthchecks.

type URLDataStore ¶

type URLDataStore interface {
	fetch.URLFetcher
	Store(*resource.WebPage) (uint64, error)
}

This interface is the contract for storing and retrieving WebPage resources. It adds a Store() method to the fetch.URLFetcher interface.

type URLString ¶

type URLString string

Type that provides the

func (URLString) Hostname ¶

func (u URLString) Hostname() string

func (URLString) String ¶

func (u URLString) String() string

type URLWithHostname ¶

type URLWithHostname interface {
	fmt.Stringer
	Hostname() string
}

net.URL and URLString both implement this interface, which is needed to generate a key for the URL.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
internal
database
sqlite This is the implementation of the store.URLDataStore interface for sqlite.	This is the implementation of the store.URLDataStore interface for sqlite.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL