Documentation
¶
Overview ¶
Package storage of the Dataflow kit describes Store interface for read/ write operations with downloaded data and parsed results.
Storage types like AWS S3, Digital Ocean Spaces, Diskv, Redis implement methods wchich satisfy Store interface.
Index ¶
Constants ¶
View Source
const ( //Storage types CACHE = "Cache" COOKIES = "Cookies" INTERMEDIATE = "Intermediate" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DiskvConn ¶
type DiskvConn struct {
// contains filtered or unexported fields
}
DiskvConn stores connection parameters for Diskv storage
type Store ¶
type Store interface {
//Reads value from storage by specified key
Read(rec Record) (value []byte, err error)
//Writes specified pair key value to storage.
//expTime value sets TTL for Redis storage.
//expTime set Metadata Expires value for S3Storage
Write(rec Record) error
//Is key expired ? It checks if parse results storage item is expired. Set up Expiration as "ITEM_EXPIRE_IN" environment variable.
//html pages cache stores this info in sResponse.Expires . It is not used for fetch endpoint.
Expired(rec Record) bool
//Delete deletes specified item from the store
Delete(rec Record) error
//DeleteAll erases all items from the store
DeleteAll() error
// Close storage connection
Close()
}
Store is the key interface of storage. All other structs implement methods wchich satisfy that interface.
Click to show internal directories.
Click to hide internal directories.