Documentation
¶
Overview ¶
Package importer is ContentKit's bulk import, POST /v1/import: how a legacy importer writes content, groups, interactions and signals. Records are idempotent by their (source, entity, external id) in content_external_ids and keep the ids they give, so re-running an import converges: changed rows are updated and nothing is duplicated. A record that fails is reported and the rest are written; sending it again resumes. Counts, group counts and search documents are rebuilt by the engine's jobs, which an import queues. Media is not imported here: importers ingest files into each version's folder through the media upload API (docs/import.md).
Index ¶
- Constants
- Variables
- type AliasRecord
- type AnswerRecord
- type Batch
- type CommentRecord
- type ContentRecord
- type EdgeRecord
- type FavoriteRecord
- type GroupRecord
- type MembershipRecord
- type MembershipSet
- type OfferRecord
- type Options
- type PollOptionRecord
- type PollRecord
- type ReactionRecord
- type RecordFailure
- type Ref
- type Result
- type Service
- type SignalRecord
- type TargetRef
- type VersionRecord
- type VoteRecord
Constants ¶
const ( SectionGroups = "groups" SectionContents = "contents" SectionMemberships = "memberships" SectionOffers = "offers" SectionAliases = "aliases" SectionComments = "comments" SectionReactions = "reactions" SectionFavorites = "favorites" SectionPolls = "polls" SectionVotes = "votes" SectionAnswers = "answers" SectionSignals = "signals" )
Sections, in the order a batch writes them.
const ( EntityContent = "content" EntityVersion = "version" EntityGroup = "group" EntityComment = "comment" EntityPoll = "poll" EntityOption = "poll_option" )
Entities of external ids.
const ( CodeInvalid = "invalid" CodeNotFound = "not_found" CodeConflict = "conflict" CodeErased = "erased" CodeInternal = "internal" )
Failure codes.
const MaxRecords = 20_000
MaxRecords bounds the records of one batch, nested versions and options included.
Variables ¶
var ErrInvalid = errors.New("importer: invalid batch")
ErrInvalid is a batch refused as a whole.
Functions ¶
This section is empty.
Types ¶
type AliasRecord ¶
type AliasRecord struct {
// Kind is the identifier space ("folder", "tag-name").
Kind string `json:"kind"`
Key string `json:"key"`
// Locator is a position inside the target, such as a page.
Locator string `json:"locator,omitempty"`
Target TargetRef `json:"target"`
}
AliasRecord resolves another system's identifier (under the batch's source) to a content item or group: a legacy URL's id, token or name.
type AnswerRecord ¶
type AnswerRecord struct {
Poll Ref `json:"poll"`
UserID string `json:"user_id"`
Text string `json:"text"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
AnswerRecord is one person's free-text answer to a poll; the classifier groups it later.
type Batch ¶
type Batch struct {
// Source names the importing system ("doujins-legacy"): external ids and
// aliases are scoped by it.
Source string `json:"source"`
// PreferenceFloor raises the preference revision sequence past a
// retained signal plane's newest revision before anything is stamped, as
// after a Postgres restore. It only ever advances.
PreferenceFloor int64 `json:"preference_floor,omitempty"`
// ProvisionScopes lets ContentKit create the AuthKit scope of an
// imported group of an auth_scope kind that names none; without it such
// a group fails.
ProvisionScopes bool `json:"provision_scopes,omitempty"`
Groups []GroupRecord `json:"groups,omitempty"`
Contents []ContentRecord `json:"contents,omitempty"`
Memberships []MembershipSet `json:"memberships,omitempty"`
Offers []OfferRecord `json:"offers,omitempty"`
Aliases []AliasRecord `json:"aliases,omitempty"`
Comments []CommentRecord `json:"comments,omitempty"`
Reactions []ReactionRecord `json:"reactions,omitempty"`
Favorites []FavoriteRecord `json:"favorites,omitempty"`
Polls []PollRecord `json:"polls,omitempty"`
Votes []VoteRecord `json:"votes,omitempty"`
Answers []AnswerRecord `json:"answers,omitempty"`
Signals []SignalRecord `json:"signals,omitempty"`
}
Batch is one import call. Its sections are written in field order, each in transactions of up to a thousand records; a record may name what an earlier section or an earlier call wrote, by id or by external id.
type CommentRecord ¶
type CommentRecord struct {
ExternalID string `json:"external_id"`
ID string `json:"id,omitempty"`
Target TargetRef `json:"target"`
// Language is the language the comment is written in; empty is unknown.
Language string `json:"language,omitempty"`
ReplyTo *Ref `json:"reply_to,omitempty"`
// UserID or, for an anonymous comment, AnonName.
UserID string `json:"user_id,omitempty"`
AnonName string `json:"anon_name,omitempty"`
Body string `json:"body"`
DeletedAt *time.Time `json:"deleted_at,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
CommentRecord is a comment on a content item (or one version), or a reply to a top-level comment on it; a reply to a reply joins its thread.
type ContentRecord ¶
type ContentRecord struct {
ExternalID string `json:"external_id"`
ID string `json:"id,omitempty"`
Kind string `json:"kind"`
// State is draft, published or deleted. A published item is live from
// PublishAt (now when omitted; scheduled when future); PublishedAt is
// when it went live.
State string `json:"state"`
PublishAt *time.Time `json:"publish_at,omitempty"`
PublishedAt *time.Time `json:"published_at,omitempty"`
// DeletedAt dates a deleted item (now when omitted).
DeletedAt *time.Time `json:"deleted_at,omitempty"`
// Listing is listed (the default), members or unlisted.
Listing string `json:"listing,omitempty"`
// Comments is open (the default), closed or hidden.
Comments string `json:"comments,omitempty"`
// ViewBaseline is the legacy system's all-time view count. Displayed
// views and the most_viewed sort add it to the viewers the signal plane
// counts; popularity and trending never see it.
ViewBaseline int64 `json:"view_baseline,omitempty"`
// Access defaults to public. A group rule names the group's id.
Access *access.Rule `json:"access,omitempty"`
Restrictions []string `json:"restrictions,omitempty"`
Attributes map[string]any `json:"attributes,omitempty"`
OwnerGroup *Ref `json:"owner_group,omitempty"`
// Cover is an inline image name of the item's media folder (i-{uuid}).
Cover string `json:"cover,omitempty"`
Creators []catalog.Creator `json:"creators,omitempty"`
Text map[string]catalog.Text `json:"text,omitempty"`
// Versions: a kind with hidden versions takes none or one.
Versions []VersionRecord `json:"versions,omitempty"`
Code string `json:"code,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
ContentRecord is a content item with its text, creators and versions. Nil text, creators, version text and segments keep what is stored; given ones replace it. Versions not listed are left as they are.
type EdgeRecord ¶
EdgeRecord is a relation from the group to another.
type FavoriteRecord ¶
type FavoriteRecord struct {
Target TargetRef `json:"target"`
UserID string `json:"user_id"`
CreatedAt *time.Time `json:"created_at,omitempty"`
}
FavoriteRecord is a person's bookmark of a content item or version.
type GroupRecord ¶
type GroupRecord struct {
ExternalID string `json:"external_id"`
// ID is kept when the external id is new (contentref.LegacyID derives a
// stable one); generated when empty.
ID string `json:"id,omitempty"`
Kind string `json:"kind"`
Slug string `json:"slug"`
// Scope is the group a scoped kind's slug is unique within.
Scope *Ref `json:"scope,omitempty"`
Parent *Ref `json:"parent,omitempty"`
// State is active (the default) or deleted.
State string `json:"state,omitempty"`
Names []groups.Name `json:"names,omitempty"`
Text []groups.Text `json:"text,omitempty"`
Attributes map[string]any `json:"attributes,omitempty"`
// CoverImage is an image of the group's media folder (i-{uuid});
// CoverContent a member whose cover it shows, imported before.
CoverImage string `json:"cover_image,omitempty"`
CoverContent *Ref `json:"cover_content,omitempty"`
// Avatar is a user-owned group's avatar: an image of its media folder.
Avatar string `json:"avatar,omitempty"`
// OwnerUserID owns a user-owned kind's group.
OwnerUserID string `json:"owner_user_id,omitempty"`
// AuthScope is the group's AuthKit scope (an auth_scope kind's); empty
// keeps the stored one.
AuthScope string `json:"auth_scope,omitempty"`
// Edges replace the group's edges to other groups.
Edges []EdgeRecord `json:"edges,omitempty"`
// Code keeps a content code the group had (9 characters); one is
// generated otherwise. A code never changes once given.
Code string `json:"code,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
GroupRecord is a group with its names and text. Nil lists keep what is stored; given ones replace it.
type MembershipRecord ¶
type MembershipRecord struct {
Group Ref `json:"group"`
Version *Ref `json:"version,omitempty"`
Role string `json:"role,omitempty"`
Position *int `json:"position,omitempty"`
ActiveFrom *time.Time `json:"active_from,omitempty"`
ActiveUntil *time.Time `json:"active_until,omitempty"`
// State is active (the default) or proposed.
State string `json:"state,omitempty"`
}
MembershipRecord places the content item, or one version of it, in a group.
type MembershipSet ¶
type MembershipSet struct {
Content Ref `json:"content"`
Kinds []string `json:"kinds"`
Memberships []MembershipRecord `json:"memberships"`
}
MembershipSet replaces a content item's memberships in the listed group kinds; memberships in other kinds stay.
type OfferRecord ¶
OfferRecord puts a content item, a version or a sellable group on sale; the offers job syncs it to the billing provider.
type Options ¶
type Options struct {
Pool *pgxpool.Pool
Schema string
Tenant string
Catalog *catalog.Store
Groups *groups.Store
Content *content.Runtime
// Offers is nil without billing: offers are skipped.
Offers *offers.Service
// Signals is nil, or answers discovery.ErrNoSignals, without a signal
// plane: signals are skipped.
Signals *discovery.Service
// Rebuild queues the jobs that rebuild what an import changed: search
// documents, group counts, listing counts, and popularity after signals.
Rebuild func(ctx context.Context, signals bool) error
Logger *slog.Logger
}
Options configures a Service over the engine's stores.
type PollOptionRecord ¶
type PollOptionRecord struct {
ExternalID string `json:"external_id"`
ID string `json:"id,omitempty"`
Label string `json:"label"`
Position int `json:"position"`
}
PollOptionRecord is one choice of a multiple-choice poll, ordered by Position.
type PollRecord ¶
type PollRecord struct {
ExternalID string `json:"external_id"`
ID string `json:"id,omitempty"`
// Kind is multiple_choice (the default) or free_text.
Kind string `json:"kind,omitempty"`
Question string `json:"question"`
Language string `json:"language,omitempty"`
// Active defaults to true.
Active *bool `json:"active,omitempty"`
LiveAt *time.Time `json:"live_at,omitempty"`
ClosesAt *time.Time `json:"closes_at,omitempty"`
DeletedAt *time.Time `json:"deleted_at,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
Options []PollOptionRecord `json:"options,omitempty"`
}
PollRecord is a site poll with its options, which replace the stored ones.
type ReactionRecord ¶
type ReactionRecord struct {
Target TargetRef `json:"target"`
UserID string `json:"user_id,omitempty"`
IP string `json:"ip,omitempty"`
Value int16 `json:"value"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
ReactionRecord is a person's like (1), dislike (-1) or neutral (0) on a content item, version, comment or group; an anonymous one is keyed by IP.
type RecordFailure ¶
type RecordFailure struct {
Section string `json:"section"`
Index int `json:"index"`
ExternalID string `json:"external_id,omitempty"`
// Code is invalid, not_found, conflict, erased or internal.
Code string `json:"code"`
Error string `json:"error"`
}
RecordFailure is one record not written. Sending it again (or the whole batch) once its cause is fixed resumes the import.
type Ref ¶
type Ref struct {
ID string `json:"id,omitempty"`
ExternalID string `json:"external_id,omitempty"`
}
Ref names a record by its ContentKit id or by its external id under the batch's source; the entity is the field's.
type Result ¶
type Result struct {
Written map[string]int `json:"written"`
// Skipped counts records an engine without the capability leaves out:
// signals without a signal plane, offers without billing.
Skipped map[string]int `json:"skipped,omitempty"`
Failed []RecordFailure `json:"failed"`
// IDs maps each written record's external id to its id, by entity.
IDs map[string]map[string]string `json:"ids"`
}
Result reports a batch: what each section wrote, the records that failed and the ids of the records with external ids.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is one tenant's bulk import.
type SignalRecord ¶
type SignalRecord struct {
Target TargetRef `json:"target"`
// UserID or, signed out, an opaque AnonKey.
UserID string `json:"user_id,omitempty"`
AnonKey string `json:"anon_key,omitempty"`
// Type is view or click.
Type string `json:"type"`
// EventID is the event's stable identity: re-sending it adds nothing.
EventID string `json:"event_id"`
Revision uint64 `json:"revision,omitempty"`
OccurredAt time.Time `json:"occurred_at"`
DurationS uint32 `json:"duration_s,omitempty"`
// Progress is pages read (reader kinds), seconds watched (player kinds)
// or percent scrolled (post kinds).
Progress uint32 `json:"progress,omitempty"`
Resume string `json:"resume,omitempty"`
}
SignalRecord is a view or click on a content item or one version, for the signal plane; skipped without one. A view on a version counts for its content too.
type TargetRef ¶
type TargetRef struct {
Content *Ref `json:"content,omitempty"`
Version *Ref `json:"version,omitempty"`
Group *Ref `json:"group,omitempty"`
Comment *Ref `json:"comment,omitempty"`
}
TargetRef is what an interaction, offer or alias is about: a content item, one version (its content is implied), a group or a comment, as the section allows.
type VersionRecord ¶
type VersionRecord struct {
ExternalID string `json:"external_id"`
ID string `json:"id,omitempty"`
Languages []string `json:"languages"`
Traits []string `json:"traits,omitempty"`
Label string `json:"label,omitempty"`
// State is published (the default), draft or deleted; times as ContentRecord's.
State string `json:"state,omitempty"`
PublishAt *time.Time `json:"publish_at,omitempty"`
PublishedAt *time.Time `json:"published_at,omitempty"`
DeletedAt *time.Time `json:"deleted_at,omitempty"`
Access *access.Rule `json:"access,omitempty"`
Attributes map[string]any `json:"attributes,omitempty"`
Ordinal *int `json:"ordinal,omitempty"`
Text map[string]catalog.VersionText `json:"text,omitempty"`
Segments []catalog.Segment `json:"segments,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
VersionRecord is one version of a content item, ordered as listed.