Documentation
¶
Overview ¶
This is the implementation of the store.URLDataStore interface for sqlite.
Use New() to make a new sqlite storage instance.
- You *must* call Open()
- The DB will be closed when the context passed to Open() is cancelled.
- Concurrent usage OK
- In-Memory DBs are supported
- The DB will be created if it doesn't exist
Index ¶
- Constants
- Variables
- func Defaults() option
- func Factory(options ...option) store.Factory
- func InMemoryDB() option
- func New(options ...option) (store.URLDataStore, error)
- func WithFile(filename string) option
- func WithFileOrEnv(filename string) option
- type AccessMode
- type JournalMode
- type Stats
- type Store
- func (s *Store) Clear() error
- func (s *Store) Create() error
- func (s Store) Fetch(url *nurl.URL) (*resource.WebPage, error)
- func (s *Store) Maintain() error
- func (s *Store) Open(ctx context.Context) error
- func (s *Store) Save(uptr *resource.WebPage) (uint64, error)
- func (s *Store) Stats() (any, error)
- type SyncMode
Constants ¶
const ( EnvDBPath = "SCRAPE_DB" FiveSecondDuration = 5 * time.Second JournalModeWAL JournalMode = "WAL" JournalModeMemory JournalMode = "MEMORY" JournalModeOff JournalMode = "OFF" BigCacheSize = 20000 NormalCacheSize = 2000 // This is actually the sqlite default SyncOff SyncMode = "OFF" SyncNormal SyncMode = "NORMAL" AccessModeRWC AccessMode = "rwc" AccessModeMemory AccessMode = "memory" )
const (
InMemoryDBName = ":memory:"
)
Variables ¶
var ( DefaultDatabase = "scrape_data/scrape.db" ErrDatabaseExists = errors.New("database already exists") ErrIsInMemory = errors.New("file path is in-memory DB (':memory:')") )
Functions ¶
func Factory ¶
Returns the factory function that can be used to instantiate a sqlite store in the cases where either creation should be delayed or where the caller may want to instantiate multiple stores with the same configuration.
func InMemoryDB ¶ added in v0.7.0
func InMemoryDB() option
func New ¶ added in v0.6.5
func New(options ...option) (store.URLDataStore, error)
func WithFileOrEnv ¶ added in v0.7.0
func WithFileOrEnv(filename string) option
Types ¶
type AccessMode ¶
type AccessMode string
type JournalMode ¶
type JournalMode string
type Stats ¶ added in v0.6.5
type Stats struct {
SqliteVersion string `json:"sqlite_version"`
PageCount int `json:"page_count"`
PageSize int `json:"page_size"`
UnusedPages int `json:"unused_pages"`
MaxPageCount int `json:"max_page_count"`
// contains filtered or unexported fields
}
func (*Stats) DatabaseSizeMB ¶ added in v0.6.5
type Store ¶ added in v0.7.0
func (*Store) Clear ¶ added in v0.7.0
Clear() will drop all tables and recreate them. This is a destructive operation. Clear uses the same query as Create(), so it will also re-create the database
func (*Store) Create ¶ added in v0.7.0
When this is called, the path to the database must already exist.
func (Store) Fetch ¶ added in v0.7.0
Fetch will return the stored data for requested URL, or nil if not found.
The returned result _may_ come from a different URL than the requested URL, if we've seen the passed URL before AND the page reported it's canonical url as being different than the requested URL.
In that case, the canonical version of the content will be returned, if we have it.
func (*Store) Open ¶ added in v0.7.0
Opens the database, creating it if it doesn't exist. The passed contexts will be used for query preparation, and to close the database when the context is cancelled.
Source Files