Documentation
¶
Overview ¶
Package export runs catalog export jobs: a selected subset (or all) of the Works emitted as MARC, N-Quads, JSON-LD, or CSV, written to the blob store, and handed back as a time-limited download link (presigned when the store signs URLs, an HMAC-token route otherwise). Jobs are records in the document store; small selections run in-request, larger ones queue for the worker loop.
Index ¶
- Constants
- Variables
- type AuthoritySelection
- type Format
- type Job
- type Selection
- type Service
- func (s *Service) Create(ctx context.Context, requester string, format Format, sel Selection) (Job, error)
- func (s *Service) CreateAuthorities(ctx context.Context, requester string, format Format, sel AuthoritySelection) (Job, error)
- func (s *Service) DownloadURL(ctx context.Context, job Job) (string, error)
- func (s *Service) Get(ctx context.Context, requester, id string, admin bool) (Job, error)
- func (s *Service) List(ctx context.Context, requester string) ([]Job, error)
- func (s *Service) Open(ctx context.Context, job Job) ([]byte, error)
- func (s *Service) Run(ctx context.Context, id string) error
- func (s *Service) RunQueued(ctx context.Context) (int, error)
- func (s *Service) Token(job Job) string
- func (s *Service) VerifyToken(job Job, token string) bool
- type Status
Constants ¶
const (
// InRequestCutoff is the largest explicit selection run synchronously.
InRequestCutoff = 200
)
Variables ¶
var ErrNotFound = errors.New("export: job not found")
ErrNotFound reports an unknown job (or one the requester may not see).
Functions ¶
This section is empty.
Types ¶
type AuthoritySelection ¶
type AuthoritySelection struct {
All bool `json:"all,omitempty"`
Vocabs []string `json:"vocabs,omitempty"` // scheme keys, e.g. ["local", "lcgft"]
Label string `json:"label,omitempty"` // label-prefix filter
}
AuthoritySelection scopes an authority export (tasks/069): everything, a vocabulary subset, and/or a label-prefix filter.
type Format ¶
type Format string
Format selects the output serialization.
const ( FormatMARC Format = "marc" // ISO 2709 via the libcodex round-trip (lossy per docs/marc-fidelity.md) FormatNQuads Format = "nquads" // corpus-style merged canonical N-Quads FormatJSONLD Format = "jsonld" // JSON-LD array via the record path (MARC-fidelity-bounded) FormatCSV Format = "csv" // projected rows (id, title, contributors, subjects, ...) )
type Job ¶
type Job struct {
ID string `json:"id"`
Requester string `json:"requester"`
Format Format `json:"format"`
Selection Selection `json:"selection"`
// Authorities, when set, makes this an authority export (tasks/069):
// the format renders terms instead of work grains.
Authorities *AuthoritySelection `json:"authorities,omitempty"`
Status Status `json:"status"`
Records int `json:"records,omitempty"`
OutputPath string `json:"outputPath,omitempty"`
Error string `json:"error,omitempty"`
CreatedAt time.Time `json:"createdAt"`
FinishedAt time.Time `json:"finishedAt,omitzero"`
// ExpiresAt bounds the download's availability (bucket lifecycle rules
// enforce the object side).
ExpiresAt time.Time `json:"expiresAt,omitzero"`
}
Job is one export request.
type Selection ¶
type Selection struct {
All bool `json:"all,omitempty"`
WorkIDs []string `json:"workIds,omitempty"`
}
Selection scopes an export: everything, or an explicit id list. Richer selections (saved queries, search results) compile down to id lists by the batch machinery (tasks/047).
type Service ¶
type Service struct {
// GrainPrefix roots the grain tree ("data/works/" under repo layout is
// implied by bibframe.GrainPath; empty prefix = paths used as-is).
GrainPrefix string
// Provider names the feed graph the CSV projection reads.
Provider string
// Vocab, when set, enables authority exports over the loaded term index
// (tasks/069).
Vocab *vocab.Index
// OrgCode is the deployment's MARC organization code; when set, MARC and
// JSON-LD exports derive each record's 040 from graph facts at decode
// time (tasks/192).
OrgCode string
// contains filtered or unexported fields
}
Service manages export jobs.
func (*Service) Create ¶
func (s *Service) Create(ctx context.Context, requester string, format Format, sel Selection) (Job, error)
Create records a new job. Explicit selections at or under InRequestCutoff run synchronously before returning; everything else stays QUEUED for the worker.
func (*Service) CreateAuthorities ¶
func (s *Service) CreateAuthorities(ctx context.Context, requester string, format Format, sel AuthoritySelection) (Job, error)
CreateAuthorities records an authority export job. Label-filtered selections run in-request (bounded); full-vocabulary dumps queue for the worker. CSV has no authority shape and is refused.
func (*Service) DownloadURL ¶
DownloadURL returns a time-limited link for a DONE job: presigned when the blob store implements blob.Signer, otherwise the token-authenticated API route.
func (*Service) Open ¶
Open streams a DONE job's output from the blob store (the fallback download route's read side).
func (*Service) Run ¶
Run executes a QUEUED job (claiming it RUNNING first, so concurrent workers cannot double-run) and streams the output into the blob store: the emitters write per grain through a pipe into PutStream, so a full-corpus export's peak memory is the per-grain working set, not the output (tasks/108). An emit failure aborts the pipe before the store commits anything. Failures land in the job's Error with StatusFailed; Run itself errors only on store problems.
func (*Service) RunQueued ¶
RunQueued drains QUEUED jobs once -- the worker-loop body for container deployments (lcatd runs it on a ticker; a Lambda deployment invokes it asynchronously).
Source Files