Help us shape the future of Go! Take the 2026 Go Developer Survey

importer

package
v0.111.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: MIT Imports: 28 Imported by: 0

Documentation

Overview

Package importer is ContentKit's bulk import, POST /v1/import: how a legacy importer writes content, groups, interactions and signals. Records are idempotent by their (source, entity, external id) in content_external_ids and keep the ids they give, so re-running an import converges: changed rows are updated and nothing is duplicated. A record that fails is reported and the rest are written; sending it again resumes. Counts, group counts and search documents are rebuilt by the engine's jobs, which an import queues. Media is not imported here: importers ingest files, covers and body images included, into each version's folders through the media upload API (docs/import.md).

Index

Constants

View Source 
const (
	SectionGroups        = "entities"
	SectionContents      = "contents"
	SectionMemberships   = "memberships"
	SectionOffers        = "offers"
	SectionAliases       = "aliases"
	SectionComments      = "comments"
	SectionReactions     = "reactions"
	SectionFavorites     = "favorites"
	SectionPolls         = "polls"
	SectionVotes         = "votes"
	SectionAnswers       = "answers"
	SectionLegacyReading = "legacy_reading"
	SectionSignals       = "signals"
	SectionDeletes       = "deletes"
)

Sections, in the order a batch writes them.

View Source 
const (
	EntityContent = "content"
	EntityVersion = "version"
	EntityGroup   = "group"
	EntityComment = "comment"
	EntityPoll    = "poll"
	EntityOption  = "poll_option"
)

Entities of external ids.

View Source 
const (
	CodeInvalid  = "invalid"
	CodeNotFound = "not_found"
	CodeConflict = "conflict"
	CodeErased   = "erased"
	CodeInternal = "internal"
)

Failure codes.

View Source 
const MaxRecords = 20_000

MaxRecords bounds the records of one batch, nested versions and options included.

Variables

View Source 
var ErrInvalid = errors.New("importer: invalid batch")

ErrInvalid is a batch refused as a whole.

Functions

This section is empty.

Types

type AliasRecord 

type AliasRecord struct {
	// Kind is the identifier space ("folder", "tag-name").
	Kind string `json:"kind"`
	Key  string `json:"key"`
	// Locator is a position inside the target, such as a page.
	Locator string    `json:"locator,omitempty"`
	Target  TargetRef `json:"target"`
}

AliasRecord resolves another system's identifier (under the batch's source) to a content item or group: a legacy URL's id, token or name.

type AnswerRecord 

type AnswerRecord struct {
	Poll      Ref        `json:"poll"`
	UserID    string     `json:"user_id"`
	Text      string     `json:"text"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

AnswerRecord is one person's free-text answer to a poll; the classifier groups it later.

type Batch 

type Batch struct {
	// Source names the importing system ("doujins-legacy"): external ids and
	// aliases are scoped by it.
	Source string `json:"source"`
	// PreferenceFloor raises the preference revision sequence past a
	// retained signal plane's newest revision before anything is stamped, as
	// after a Postgres restore. It only ever advances.
	PreferenceFloor int64 `json:"preference_floor,omitempty"`
	// ProvisionScopes lets ContentKit create the AuthKit scope of an
	// imported group of an auth_scope kind that names none; without it such
	// a group fails.
	ProvisionScopes bool                  `json:"provision_scopes,omitempty"`
	Groups          []GroupRecord         `json:"entities,omitempty"`
	Contents        []ContentRecord       `json:"contents,omitempty"`
	Memberships     []MembershipSet       `json:"memberships,omitempty"`
	Offers          []OfferRecord         `json:"offers,omitempty"`
	Aliases         []AliasRecord         `json:"aliases,omitempty"`
	Comments        []CommentRecord       `json:"comments,omitempty"`
	Reactions       []ReactionRecord      `json:"reactions,omitempty"`
	Favorites       []FavoriteRecord      `json:"favorites,omitempty"`
	Polls           []PollRecord          `json:"polls,omitempty"`
	Votes           []VoteRecord          `json:"votes,omitempty"`
	Answers         []AnswerRecord        `json:"answers,omitempty"`
	LegacyReading   []LegacyReadingRecord `json:"legacy_reading,omitempty"`
	Signals         []SignalRecord        `json:"signals,omitempty"`
	// Deletes, written last, remove records gone from the source. Records
	// are never deleted by their absence from a batch.
	Deletes []DeleteRecord `json:"deletes,omitempty"`
}

Batch is one import call. Its sections are written in field order, each in transactions of up to a thousand records; a record may name what an earlier section or an earlier call wrote, by id or by external id.

func (Batch) RecordCount added in v0.111.1 

func (b Batch) RecordCount() int

RecordCount returns the count checked against MaxRecords, including nested content versions and poll options. Importers can use it to size their batches.

type CommentRecord 

type CommentRecord struct {
	ExternalID string    `json:"external_id"`
	ID         string    `json:"id,omitempty"`
	Target     TargetRef `json:"target"`
	// Language is the language the comment is written in; empty is unknown.
	Language string `json:"language,omitempty"`
	ReplyTo  *Ref   `json:"reply_to,omitempty"`
	// UserID or, for an anonymous comment, AnonName.
	UserID    string     `json:"user_id,omitempty"`
	AnonName  string     `json:"anon_name,omitempty"`
	Body      string     `json:"body"`
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

CommentRecord is a comment on a content item (or one version), or a reply to a top-level comment on it; a reply to a reply joins its thread.

type ContentRecord 

type ContentRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Kind       string `json:"kind"`
	// State is draft, published or deleted. A published item is live from
	// PublishAt (now when omitted; scheduled when future); PublishedAt is
	// when it went live.
	State       string     `json:"state"`
	PublishAt   *time.Time `json:"publish_at,omitempty"`
	PublishedAt *time.Time `json:"published_at,omitempty"`
	// DeletedAt dates a deleted item (now when omitted).
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	// Listing is listed (the default), members or unlisted.
	Listing string `json:"listing,omitempty"`
	// Comments is open (the default), closed or hidden.
	Comments string `json:"comments,omitempty"`
	// Access defaults to public. A group rule names the group's id.
	Access       *access.Rule            `json:"access,omitempty"`
	Restrictions []string                `json:"restrictions,omitempty"`
	Attributes   map[string]any          `json:"attributes,omitempty"`
	OwnerGroup   *Ref                    `json:"owner_group,omitempty"`
	Creators     []catalog.Creator       `json:"creators,omitempty"`
	Text         map[string]catalog.Text `json:"text,omitempty"`
	// Versions: a kind with hidden versions takes none or one.
	Versions  []VersionRecord `json:"versions,omitempty"`
	Code      string          `json:"code,omitempty"`
	CreatedAt *time.Time      `json:"created_at,omitempty"`
	UpdatedAt *time.Time      `json:"updated_at,omitempty"`
}

ContentRecord is a content item with its text, creators and versions. Nil text, creators, version text and segments keep what is stored; given ones replace it. Versions not listed are left as they are.

type DeleteRecord added in v0.85.0 

type DeleteRecord struct {
	// Entity is content, version, group, comment or poll.
	Entity     string `json:"entity"`
	ExternalID string `json:"external_id"`
}

DeleteRecord deletes a record the source no longer has, as its delete route would: a content item, version, group, comment or poll, by its external id under the batch's source. One already deleted (or since erased) stays as it is, so re-sending converges; an external id never imported fails not_found.

type EdgeRecord 

type EdgeRecord struct {
	Relation string `json:"relation"`
	To       Ref    `json:"to"`
}

EdgeRecord is a relation from the group to another.

type ExternalID added in v0.86.0 

type ExternalID struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id"`
	Deleted    bool   `json:"deleted"`
}

ExternalID is one imported record and its id. Deleted: ContentKit holds it deleted or no longer holds it, so it needs no delete record.

type ExternalIDList added in v0.86.0 

type ExternalIDList struct {
	// Entity is content, version, group, comment, poll or poll_option; each
	// at most once per query.
	Entity string `json:"entity"`
	// Cursor is the previous page's next_cursor.
	Cursor string `json:"cursor,omitempty"`
	// Limit defaults to 1000 and is capped at 10,000.
	Limit int `json:"limit,omitempty"`
}

ExternalIDList asks one page of an entity's external ids.

type ExternalIDPage added in v0.86.0 

type ExternalIDPage struct {
	Entity string       `json:"entity"`
	IDs    []ExternalID `json:"ids"`
	// NextCursor pages on; empty on the last page.
	NextCursor string `json:"next_cursor,omitempty"`
}

ExternalIDPage is one entity's page, in external id order.

type ExternalIDs added in v0.86.0 

type ExternalIDs struct {
	Lists []ExternalIDPage `json:"lists"`
}

ExternalIDs answers an ExternalIDsQuery: a page per list, in order.

type ExternalIDsQuery added in v0.86.0 

type ExternalIDsQuery struct {
	Source string           `json:"source"`
	Lists  []ExternalIDList `json:"lists"`
}

ExternalIDsQuery asks which external ids imports under Source hold, one cursor-paged list per entity: what a persistent sync compares with its source to find the deletes to send.

type FavoriteRecord 

type FavoriteRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

FavoriteRecord is a person's bookmark of a content item or version.

type GroupRecord 

type GroupRecord struct {
	ExternalID string `json:"external_id"`
	// ID is kept when the external id is new (contentref.LegacyID derives a
	// stable one); generated when empty.
	ID   string `json:"id,omitempty"`
	Kind string `json:"kind"`
	Slug string `json:"slug"`
	// Internal classifications can drive domain policy but never public metadata.
	// Omission preserves an existing entity's classification; new entities are public.
	Internal *bool `json:"internal,omitempty"`
	// Scope is the group a scoped kind's slug is unique within.
	Scope  *Ref `json:"scope,omitempty"`
	Parent *Ref `json:"parent,omitempty"`
	// State is active (the default) or deleted.
	State      string         `json:"state,omitempty"`
	Names      []groups.Name  `json:"names,omitempty"`
	Text       []groups.Text  `json:"text,omitempty"`
	Attributes map[string]any `json:"attributes,omitempty"`
	// CoverImage is an image of the group's media folder (i-{uuid});
	// CoverContent a member whose cover it shows, imported before.
	CoverImage   string `json:"cover_image,omitempty"`
	CoverContent *Ref   `json:"cover_content,omitempty"`
	// Avatar is a user-owned group's avatar: an image of its media folder.
	Avatar string `json:"avatar,omitempty"`
	// OwnerUserID owns a user-owned kind's group.
	OwnerUserID string `json:"owner_user_id,omitempty"`
	// AuthScope is the group's AuthKit scope (an auth_scope kind's); empty
	// keeps the stored one.
	AuthScope string `json:"auth_scope,omitempty"`
	// Edges replace the group's edges to other groups.
	Edges []EdgeRecord `json:"edges,omitempty"`
	// Code keeps a content code the group had (9 characters); one is
	// generated otherwise. A code never changes once given.
	Code      string     `json:"code,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

GroupRecord is a group with its names and text. Nil lists keep what is stored; given ones replace it.

type LegacyReadingRecord added in v0.104.0 

type LegacyReadingRecord struct {
	Target      TargetRef  `json:"target"`
	UserID      string     `json:"user_id,omitempty"`
	AnonKey     string     `json:"anon_key,omitempty"`
	FirstReadAt *time.Time `json:"first_read_at,omitempty"`
	LastReadAt  time.Time  `json:"last_read_at"`
	Progress    uint32     `json:"progress"`
	ProgressMax uint32     `json:"progress_max"`
	Completed   bool       `json:"completed"`
	Resume      string     `json:"resume,omitempty"`
	Revision    uint64     `json:"revision,omitempty"`
}

LegacyReadingRecord preserves source bookmarks/progress as legacy state, never as measured attention or popularity evidence.

type MembershipRecord 

type MembershipRecord struct {
	Group       Ref        `json:"group"`
	Version     *Ref       `json:"version,omitempty"`
	Role        string     `json:"role,omitempty"`
	Position    *int       `json:"position,omitempty"`
	ActiveFrom  *time.Time `json:"active_from,omitempty"`
	ActiveUntil *time.Time `json:"active_until,omitempty"`
	// State is active (the default) or proposed.
	State string `json:"state,omitempty"`
}

MembershipRecord places the content item, or one version of it, in a group.

type MembershipSet 

type MembershipSet struct {
	Content     Ref                `json:"content"`
	Kinds       []string           `json:"kinds"`
	Memberships []MembershipRecord `json:"memberships"`
}

MembershipSet replaces a content item's memberships in the listed group kinds; memberships in other kinds stay.

type OfferRecord 

type OfferRecord struct {
	Target TargetRef    `json:"target"`
	Terms  offers.Terms `json:"terms"`
}

OfferRecord puts a content item, a version or a sellable group on sale; the offers job syncs it to the billing provider.

type Options 

type Options struct {
	Pool    *pgxpool.Pool
	Schema  string
	Tenant  string
	Catalog *catalog.Store
	Groups  *groups.Store
	Content *content.Runtime
	// Offers is nil without billing: offers are skipped.
	Offers *offers.Service
	// Signals is nil, or answers discovery.ErrNoSignals, without a signal
	// plane: signals are skipped.
	Signals *discovery.Service
	// Rebuild queues the jobs that rebuild what an import changed: search
	// documents, group counts and listing counts.
	Rebuild func(ctx context.Context) error
	Logger  *slog.Logger
}

Options configures a Service over the engine's stores.

type PollOptionRecord 

type PollOptionRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Label      string `json:"label"`
	Position   int    `json:"position"`
}

PollOptionRecord is one choice of a multiple-choice poll, ordered by Position.

type PollRecord 

type PollRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	// Kind is multiple_choice (the default) or free_text.
	Kind     string `json:"kind,omitempty"`
	Question string `json:"question"`
	Language string `json:"language,omitempty"`
	// Active defaults to true.
	Active    *bool              `json:"active,omitempty"`
	LiveAt    *time.Time         `json:"live_at,omitempty"`
	ClosesAt  *time.Time         `json:"closes_at,omitempty"`
	DeletedAt *time.Time         `json:"deleted_at,omitempty"`
	CreatedAt *time.Time         `json:"created_at,omitempty"`
	UpdatedAt *time.Time         `json:"updated_at,omitempty"`
	Options   []PollOptionRecord `json:"options,omitempty"`
}

PollRecord is a site poll with its options, which replace the stored ones.

type ReactionRecord 

type ReactionRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	Value     int16      `json:"value"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

ReactionRecord is a person's like (1), dislike (-1) or neutral (0) on a content item, version, comment or group; an anonymous one is keyed by IP.

type RecordFailure 

type RecordFailure struct {
	Section    string `json:"section"`
	Index      int    `json:"index"`
	ExternalID string `json:"external_id,omitempty"`
	// Code is invalid, not_found, conflict, erased or internal.
	Code  string `json:"code"`
	Error string `json:"error"`
}

RecordFailure is one record not written. Sending it again (or the whole batch) once its cause is fixed resumes the import.

type Ref 

type Ref struct {
	ID         string `json:"id,omitempty"`
	ExternalID string `json:"external_id,omitempty"`
}

Ref names a record by its ContentKit id or by its external id under the batch's source; the entity is the field's.

type Result 

type Result struct {
	Written map[string]int `json:"written"`
	// Skipped counts records an engine without the capability leaves out:
	// signals without a signal plane, offers without billing.
	Skipped map[string]int  `json:"skipped,omitempty"`
	Failed  []RecordFailure `json:"failed"`
	// IDs maps each written record's external id to its id, by entity.
	IDs map[string]map[string]string `json:"ids"`
}

Result reports a batch: what each section wrote, the records that failed and the ids of the records with external ids.

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service is one tenant's bulk import.

func New 

func New(o Options) (*Service, error)

New checks the options.

func (*Service) ExternalIDs added in v0.86.0 

func (s *Service) ExternalIDs(ctx context.Context, q ExternalIDsQuery) (ExternalIDs, error)

ExternalIDs pages the external ids imports under q.Source hold, per entity.

func (*Service) Handler 

func (s *Service) Handler(a auth.Authenticator) (http.Handler, error)

Handler serves POST /v1/import and /v1/import/external-ids to holders of access.ImportWrite.

func (*Service) Import 

func (s *Service) Import(ctx context.Context, b Batch) (Result, error)

Import writes a batch, section by section. It fails as a whole only for a batch it refuses outright or when ctx ends; records that fail are in the result.

type SignalRecord 

type SignalRecord struct {
	Target TargetRef `json:"target"`
	// UserID or, signed out, an opaque AnonKey.
	UserID  string `json:"user_id,omitempty"`
	AnonKey string `json:"anon_key,omitempty"`
	// Type is click; imported raw opens are rejected.
	Type string `json:"type"`
	// EventID is the event's stable identity: re-sending it adds nothing.
	EventID    string    `json:"event_id"`
	Revision   uint64    `json:"revision,omitempty"`
	OccurredAt time.Time `json:"occurred_at"`
}

SignalRecord imports an explicit click. Existing bookmarks/progress use LegacyReadingRecord; raw opens and imported counters are not accepted.

type TargetRef 

type TargetRef struct {
	Content *Ref `json:"content,omitempty"`
	Version *Ref `json:"version,omitempty"`
	Group   *Ref `json:"group,omitempty"`
	Comment *Ref `json:"comment,omitempty"`
}

TargetRef is what an interaction, offer or alias is about: a content item, one version (its content is implied), a group or a comment, as the section allows.

type VersionRecord 

type VersionRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	// Code is the version code (?v=) to keep, as Code is the content's: empty
	// keeps the stored one or assigns one.
	Code      string   `json:"code,omitempty"`
	Languages []string `json:"languages"`
	Traits    []string `json:"traits,omitempty"`
	Label     string   `json:"label,omitempty"`
	// State is published (the default), draft or deleted; times as ContentRecord's.
	State       string         `json:"state,omitempty"`
	PublishAt   *time.Time     `json:"publish_at,omitempty"`
	PublishedAt *time.Time     `json:"published_at,omitempty"`
	DeletedAt   *time.Time     `json:"deleted_at,omitempty"`
	Access      *access.Rule   `json:"access,omitempty"`
	Attributes  map[string]any `json:"attributes,omitempty"`
	Ordinal     *int           `json:"ordinal,omitempty"`
	// Cover selects one of the version's cover uploads (i-{uuid}): a
	// record's one legacy cover is uploaded to each of its versions. Left
	// out, the stored selection stays.
	Cover     string                         `json:"cover,omitempty"`
	Text      map[string]catalog.VersionText `json:"text,omitempty"`
	Segments  []catalog.Segment              `json:"segments,omitempty"`
	CreatedAt *time.Time                     `json:"created_at,omitempty"`
	UpdatedAt *time.Time                     `json:"updated_at,omitempty"`
}

VersionRecord is one version of a content item, ordered as listed.

type VoteRecord 

type VoteRecord struct {
	Poll      Ref        `json:"poll"`
	Option    Ref        `json:"option"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

VoteRecord is one person's (or, anonymous, one IP's) choice in a poll.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.