importer

package
v0.83.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package importer is ContentKit's bulk import, POST /v1/import: how a legacy importer writes content, groups, interactions and signals. Records are idempotent by their (source, entity, external id) in content_external_ids and keep the ids they give, so re-running an import converges: changed rows are updated and nothing is duplicated. A record that fails is reported and the rest are written; sending it again resumes. Counts, group counts and search documents are rebuilt by the engine's jobs, which an import queues. Media is not imported here: importers ingest files into each version's folder through the media upload API (docs/import.md).

Index

Constants

View Source
const (
	SectionGroups      = "groups"
	SectionContents    = "contents"
	SectionMemberships = "memberships"
	SectionOffers      = "offers"
	SectionAliases     = "aliases"
	SectionComments    = "comments"
	SectionReactions   = "reactions"
	SectionFavorites   = "favorites"
	SectionPolls       = "polls"
	SectionVotes       = "votes"
	SectionAnswers     = "answers"
	SectionSignals     = "signals"
)

Sections, in the order a batch writes them.

View Source
const (
	EntityContent = "content"
	EntityVersion = "version"
	EntityGroup   = "group"
	EntityComment = "comment"
	EntityPoll    = "poll"
	EntityOption  = "poll_option"
)

Entities of external ids.

View Source
const (
	CodeInvalid  = "invalid"
	CodeNotFound = "not_found"
	CodeConflict = "conflict"
	CodeErased   = "erased"
	CodeInternal = "internal"
)

Failure codes.

View Source
const MaxRecords = 20_000

MaxRecords bounds the records of one batch, nested versions and options included.

Variables

View Source
var ErrInvalid = errors.New("importer: invalid batch")

ErrInvalid is a batch refused as a whole.

Functions

This section is empty.

Types

type AliasRecord

type AliasRecord struct {
	// Kind is the identifier space ("folder", "tag-name").
	Kind string `json:"kind"`
	Key  string `json:"key"`
	// Locator is a position inside the target, such as a page.
	Locator string    `json:"locator,omitempty"`
	Target  TargetRef `json:"target"`
}

AliasRecord resolves another system's identifier (under the batch's source) to a content item or group: a legacy URL's id, token or name.

type AnswerRecord

type AnswerRecord struct {
	Poll      Ref        `json:"poll"`
	UserID    string     `json:"user_id"`
	Text      string     `json:"text"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

AnswerRecord is one person's free-text answer to a poll; the classifier groups it later.

type Batch

type Batch struct {
	// Source names the importing system ("doujins-legacy"): external ids and
	// aliases are scoped by it.
	Source string `json:"source"`
	// PreferenceFloor raises the preference revision sequence past a
	// retained signal plane's newest revision before anything is stamped, as
	// after a Postgres restore. It only ever advances.
	PreferenceFloor int64 `json:"preference_floor,omitempty"`
	// ProvisionScopes lets ContentKit create the AuthKit scope of an
	// imported group of an auth_scope kind that names none; without it such
	// a group fails.
	ProvisionScopes bool             `json:"provision_scopes,omitempty"`
	Groups          []GroupRecord    `json:"groups,omitempty"`
	Contents        []ContentRecord  `json:"contents,omitempty"`
	Memberships     []MembershipSet  `json:"memberships,omitempty"`
	Offers          []OfferRecord    `json:"offers,omitempty"`
	Aliases         []AliasRecord    `json:"aliases,omitempty"`
	Comments        []CommentRecord  `json:"comments,omitempty"`
	Reactions       []ReactionRecord `json:"reactions,omitempty"`
	Favorites       []FavoriteRecord `json:"favorites,omitempty"`
	Polls           []PollRecord     `json:"polls,omitempty"`
	Votes           []VoteRecord     `json:"votes,omitempty"`
	Answers         []AnswerRecord   `json:"answers,omitempty"`
	Signals         []SignalRecord   `json:"signals,omitempty"`
}

Batch is one import call. Its sections are written in field order, each in transactions of up to a thousand records; a record may name what an earlier section or an earlier call wrote, by id or by external id.

type CommentRecord

type CommentRecord struct {
	ExternalID string    `json:"external_id"`
	ID         string    `json:"id,omitempty"`
	Target     TargetRef `json:"target"`
	// Language is the language the comment is written in; empty is unknown.
	Language string `json:"language,omitempty"`
	ReplyTo  *Ref   `json:"reply_to,omitempty"`
	// UserID or, for an anonymous comment, AnonName.
	UserID    string     `json:"user_id,omitempty"`
	AnonName  string     `json:"anon_name,omitempty"`
	Body      string     `json:"body"`
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

CommentRecord is a comment on a content item (or one version), or a reply to a top-level comment on it; a reply to a reply joins its thread.

type ContentRecord

type ContentRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Kind       string `json:"kind"`
	// State is draft, published or deleted. A published item is live from
	// PublishAt (now when omitted; scheduled when future); PublishedAt is
	// when it went live.
	State       string     `json:"state"`
	PublishAt   *time.Time `json:"publish_at,omitempty"`
	PublishedAt *time.Time `json:"published_at,omitempty"`
	// DeletedAt dates a deleted item (now when omitted).
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	// Listing is listed (the default), members or unlisted.
	Listing string `json:"listing,omitempty"`
	// Comments is open (the default), closed or hidden.
	Comments string `json:"comments,omitempty"`
	// ViewBaseline is the legacy system's all-time view count. Displayed
	// views and the most_viewed sort add it to the viewers the signal plane
	// counts; popularity and trending never see it.
	ViewBaseline int64 `json:"view_baseline,omitempty"`
	// Access defaults to public. A group rule names the group's id.
	Access       *access.Rule   `json:"access,omitempty"`
	Restrictions []string       `json:"restrictions,omitempty"`
	Attributes   map[string]any `json:"attributes,omitempty"`
	OwnerGroup   *Ref           `json:"owner_group,omitempty"`
	// Cover is an inline image name of the item's media folder (i-{uuid}).
	Cover    string                  `json:"cover,omitempty"`
	Creators []catalog.Creator       `json:"creators,omitempty"`
	Text     map[string]catalog.Text `json:"text,omitempty"`
	// Versions: a kind with hidden versions takes none or one.
	Versions  []VersionRecord `json:"versions,omitempty"`
	Code      string          `json:"code,omitempty"`
	CreatedAt *time.Time      `json:"created_at,omitempty"`
	UpdatedAt *time.Time      `json:"updated_at,omitempty"`
}

ContentRecord is a content item with its text, creators and versions. Nil text, creators, version text and segments keep what is stored; given ones replace it. Versions not listed are left as they are.

type EdgeRecord

type EdgeRecord struct {
	Relation string `json:"relation"`
	To       Ref    `json:"to"`
}

EdgeRecord is a relation from the group to another.

type FavoriteRecord

type FavoriteRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

FavoriteRecord is a person's bookmark of a content item or version.

type GroupRecord

type GroupRecord struct {
	ExternalID string `json:"external_id"`
	// ID is kept when the external id is new (contentref.LegacyID derives a
	// stable one); generated when empty.
	ID   string `json:"id,omitempty"`
	Kind string `json:"kind"`
	Slug string `json:"slug"`
	// Scope is the group a scoped kind's slug is unique within.
	Scope  *Ref `json:"scope,omitempty"`
	Parent *Ref `json:"parent,omitempty"`
	// State is active (the default) or deleted.
	State      string         `json:"state,omitempty"`
	Names      []groups.Name  `json:"names,omitempty"`
	Text       []groups.Text  `json:"text,omitempty"`
	Attributes map[string]any `json:"attributes,omitempty"`
	// CoverImage is an image of the group's media folder (i-{uuid});
	// CoverContent a member whose cover it shows, imported before.
	CoverImage   string `json:"cover_image,omitempty"`
	CoverContent *Ref   `json:"cover_content,omitempty"`
	// Avatar is a user-owned group's avatar: an image of its media folder.
	Avatar string `json:"avatar,omitempty"`
	// OwnerUserID owns a user-owned kind's group.
	OwnerUserID string `json:"owner_user_id,omitempty"`
	// AuthScope is the group's AuthKit scope (an auth_scope kind's); empty
	// keeps the stored one.
	AuthScope string `json:"auth_scope,omitempty"`
	// Edges replace the group's edges to other groups.
	Edges []EdgeRecord `json:"edges,omitempty"`
	// Code keeps a content code the group had (9 characters); one is
	// generated otherwise. A code never changes once given.
	Code      string     `json:"code,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

GroupRecord is a group with its names and text. Nil lists keep what is stored; given ones replace it.

type MembershipRecord

type MembershipRecord struct {
	Group       Ref        `json:"group"`
	Version     *Ref       `json:"version,omitempty"`
	Role        string     `json:"role,omitempty"`
	Position    *int       `json:"position,omitempty"`
	ActiveFrom  *time.Time `json:"active_from,omitempty"`
	ActiveUntil *time.Time `json:"active_until,omitempty"`
	// State is active (the default) or proposed.
	State string `json:"state,omitempty"`
}

MembershipRecord places the content item, or one version of it, in a group.

type MembershipSet

type MembershipSet struct {
	Content     Ref                `json:"content"`
	Kinds       []string           `json:"kinds"`
	Memberships []MembershipRecord `json:"memberships"`
}

MembershipSet replaces a content item's memberships in the listed group kinds; memberships in other kinds stay.

type OfferRecord

type OfferRecord struct {
	Target TargetRef    `json:"target"`
	Terms  offers.Terms `json:"terms"`
}

OfferRecord puts a content item, a version or a sellable group on sale; the offers job syncs it to the billing provider.

type Options

type Options struct {
	Pool    *pgxpool.Pool
	Schema  string
	Tenant  string
	Catalog *catalog.Store
	Groups  *groups.Store
	Content *content.Runtime
	// Offers is nil without billing: offers are skipped.
	Offers *offers.Service
	// Signals is nil, or answers discovery.ErrNoSignals, without a signal
	// plane: signals are skipped.
	Signals *discovery.Service
	// Rebuild queues the jobs that rebuild what an import changed: search
	// documents, group counts, listing counts, and popularity after signals.
	Rebuild func(ctx context.Context, signals bool) error
	Logger  *slog.Logger
}

Options configures a Service over the engine's stores.

type PollOptionRecord

type PollOptionRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Label      string `json:"label"`
	Position   int    `json:"position"`
}

PollOptionRecord is one choice of a multiple-choice poll, ordered by Position.

type PollRecord

type PollRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	// Kind is multiple_choice (the default) or free_text.
	Kind     string `json:"kind,omitempty"`
	Question string `json:"question"`
	Language string `json:"language,omitempty"`
	// Active defaults to true.
	Active    *bool              `json:"active,omitempty"`
	LiveAt    *time.Time         `json:"live_at,omitempty"`
	ClosesAt  *time.Time         `json:"closes_at,omitempty"`
	DeletedAt *time.Time         `json:"deleted_at,omitempty"`
	CreatedAt *time.Time         `json:"created_at,omitempty"`
	UpdatedAt *time.Time         `json:"updated_at,omitempty"`
	Options   []PollOptionRecord `json:"options,omitempty"`
}

PollRecord is a site poll with its options, which replace the stored ones.

type ReactionRecord

type ReactionRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	Value     int16      `json:"value"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

ReactionRecord is a person's like (1), dislike (-1) or neutral (0) on a content item, version, comment or group; an anonymous one is keyed by IP.

type RecordFailure

type RecordFailure struct {
	Section    string `json:"section"`
	Index      int    `json:"index"`
	ExternalID string `json:"external_id,omitempty"`
	// Code is invalid, not_found, conflict, erased or internal.
	Code  string `json:"code"`
	Error string `json:"error"`
}

RecordFailure is one record not written. Sending it again (or the whole batch) once its cause is fixed resumes the import.

type Ref

type Ref struct {
	ID         string `json:"id,omitempty"`
	ExternalID string `json:"external_id,omitempty"`
}

Ref names a record by its ContentKit id or by its external id under the batch's source; the entity is the field's.

type Result

type Result struct {
	Written map[string]int `json:"written"`
	// Skipped counts records an engine without the capability leaves out:
	// signals without a signal plane, offers without billing.
	Skipped map[string]int  `json:"skipped,omitempty"`
	Failed  []RecordFailure `json:"failed"`
	// IDs maps each written record's external id to its id, by entity.
	IDs map[string]map[string]string `json:"ids"`
}

Result reports a batch: what each section wrote, the records that failed and the ids of the records with external ids.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service is one tenant's bulk import.

func New

func New(o Options) (*Service, error)

New checks the options.

func (*Service) Handler

func (s *Service) Handler(a auth.Authenticator) (http.Handler, error)

Handler serves POST /v1/import to holders of access.ImportWrite.

func (*Service) Import

func (s *Service) Import(ctx context.Context, b Batch) (Result, error)

Import writes a batch, section by section. It fails as a whole only for a batch it refuses outright or when ctx ends; records that fail are in the result.

type SignalRecord

type SignalRecord struct {
	Target TargetRef `json:"target"`
	// UserID or, signed out, an opaque AnonKey.
	UserID  string `json:"user_id,omitempty"`
	AnonKey string `json:"anon_key,omitempty"`
	// Type is view or click.
	Type string `json:"type"`
	// EventID is the event's stable identity: re-sending it adds nothing.
	EventID    string    `json:"event_id"`
	Revision   uint64    `json:"revision,omitempty"`
	OccurredAt time.Time `json:"occurred_at"`
	DurationS  uint32    `json:"duration_s,omitempty"`
	// Progress is pages read (reader kinds), seconds watched (player kinds)
	// or percent scrolled (post kinds).
	Progress uint32 `json:"progress,omitempty"`
	Resume   string `json:"resume,omitempty"`
}

SignalRecord is a view or click on a content item or one version, for the signal plane; skipped without one. A view on a version counts for its content too.

type TargetRef

type TargetRef struct {
	Content *Ref `json:"content,omitempty"`
	Version *Ref `json:"version,omitempty"`
	Group   *Ref `json:"group,omitempty"`
	Comment *Ref `json:"comment,omitempty"`
}

TargetRef is what an interaction, offer or alias is about: a content item, one version (its content is implied), a group or a comment, as the section allows.

type VersionRecord

type VersionRecord struct {
	ExternalID string   `json:"external_id"`
	ID         string   `json:"id,omitempty"`
	Languages  []string `json:"languages"`
	Traits     []string `json:"traits,omitempty"`
	Label      string   `json:"label,omitempty"`
	// State is published (the default), draft or deleted; times as ContentRecord's.
	State       string                         `json:"state,omitempty"`
	PublishAt   *time.Time                     `json:"publish_at,omitempty"`
	PublishedAt *time.Time                     `json:"published_at,omitempty"`
	DeletedAt   *time.Time                     `json:"deleted_at,omitempty"`
	Access      *access.Rule                   `json:"access,omitempty"`
	Attributes  map[string]any                 `json:"attributes,omitempty"`
	Ordinal     *int                           `json:"ordinal,omitempty"`
	Text        map[string]catalog.VersionText `json:"text,omitempty"`
	Segments    []catalog.Segment              `json:"segments,omitempty"`
	CreatedAt   *time.Time                     `json:"created_at,omitempty"`
	UpdatedAt   *time.Time                     `json:"updated_at,omitempty"`
}

VersionRecord is one version of a content item, ordered as listed.

type VoteRecord

type VoteRecord struct {
	Poll      Ref        `json:"poll"`
	Option    Ref        `json:"option"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

VoteRecord is one person's (or, anonymous, one IP's) choice in a poll.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL