Help us shape the future of Go! Take the 2026 Go Developer Survey

importer

package
v0.131.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 26, 2026 License: MIT Imports: 28 Imported by: 0

Documentation

Overview

Package importer is ContentKit's bulk import, POST /v1/import: how a legacy importer writes content, entities, interactions and signals. Records are idempotent by their (source, entity, external id) in content_external_ids and keep the ids they give, so re-running an import converges: changed rows are updated and nothing is duplicated. A record that fails is reported and the rest are written; sending it again resumes. Counts, entity counts and search documents are rebuilt by the engine's jobs, which an import queues. Media is not imported here: importers ingest files, covers and body images included, into each version's folders through the media upload API (docs/import.md).

Index

Constants

View Source 
const (
	SectionChannels      = "channels"
	SectionEntities      = "entities"
	SectionContents      = "contents"
	SectionMemberships   = "memberships"
	SectionAliases       = "aliases"
	SectionComments      = "comments"
	SectionReactions     = "reactions"
	SectionFavorites     = "favorites"
	SectionPolls         = "polls"
	SectionVotes         = "votes"
	SectionLegacyReading = "legacy_reading"
	SectionSignals       = "signals"
	SectionDeletes       = "deletes"
)

Sections, in the order a batch writes them.

View Source 
const (
	RecordChannel = "channel"
	RecordContent = "content"
	RecordVersion = "version"
	RecordEntity  = "group"
	RecordComment = "comment"
	RecordPoll    = "poll"
	RecordOption  = "poll_option"
)

Entities of external ids.

View Source 
const (
	CodeInvalid  = "invalid"
	CodeNotFound = "not_found"
	CodeConflict = "conflict"
	CodeErased   = "erased"
	CodeInternal = "internal"
)

Failure codes.

View Source 
const MaxRecords = 20_000

MaxRecords bounds the records of one batch, nested versions and options included.

Variables

View Source 
var ErrInvalid = errors.New("importer: invalid batch")

ErrInvalid is a batch refused as a whole.

Functions

This section is empty.

Types

type AliasRecord 

type AliasRecord struct {
	// Kind is the identifier space ("folder", "tag-name").
	Kind string `json:"kind"`
	Key  string `json:"key"`
	// Locator is a position inside the target, such as a page.
	Locator string    `json:"locator,omitempty"`
	Target  TargetRef `json:"target"`
}

AliasRecord resolves another system's identifier (under the batch's source) to a content item or entity: a legacy URL's id, token or name.

type Batch 

type Batch struct {
	// Source names the importing system ("doujins-legacy"): external ids and
	// aliases are scoped by it.
	Source string `json:"source"`
	// PreferenceFloor raises the preference revision sequence past a
	// retained signal plane's newest revision before anything is stamped, as
	// after a Postgres restore. It only ever advances.
	PreferenceFloor int64 `json:"preference_floor,omitempty"`
	// ProvisionScopes lets ContentKit create a missing imported channel's
	// AuthKit scope. It never grants authority through a metadata entity.
	ProvisionScopes bool                  `json:"provision_scopes,omitempty"`
	Entities        []EntityRecord        `json:"entities,omitempty"`
	Channels        []ChannelRecord       `json:"channels,omitempty"`
	Contents        []ContentRecord       `json:"contents,omitempty"`
	Memberships     []MembershipSet       `json:"memberships,omitempty"`
	Aliases         []AliasRecord         `json:"aliases,omitempty"`
	Comments        []CommentRecord       `json:"comments,omitempty"`
	Reactions       []ReactionRecord      `json:"reactions,omitempty"`
	Favorites       []FavoriteRecord      `json:"favorites,omitempty"`
	Polls           []PollRecord          `json:"polls,omitempty"`
	Votes           []VoteRecord          `json:"votes,omitempty"`
	LegacyReading   []LegacyReadingRecord `json:"legacy_reading,omitempty"`
	Signals         []SignalRecord        `json:"signals,omitempty"`
	// Deletes, written last, remove records gone from the source. Records
	// are never deleted by their absence from a batch.
	Deletes []DeleteRecord `json:"deletes,omitempty"`
}

Batch is one import call. Its sections are written in field order, each in transactions of up to a thousand records; a record may name what an earlier section or an earlier call wrote, by id or by external id.

func (Batch) RecordCount added in v0.111.1 

func (b Batch) RecordCount() int

RecordCount returns the count checked against MaxRecords, including nested content versions and poll options. Importers can use it to size their batches.

type ChannelRecord added in v0.131.0 

type ChannelRecord struct {
	ExternalID   string          `json:"external_id"`
	ID           string          `json:"id,omitempty"`
	Slug         string          `json:"slug"`
	State        string          `json:"state,omitempty"`
	Names        []entities.Name `json:"names,omitempty"`
	Text         []entities.Text `json:"text,omitempty"`
	CoverImage   string          `json:"cover_image,omitempty"`
	CoverContent *Ref            `json:"cover_content,omitempty"`
	Avatar       string          `json:"avatar,omitempty"`
	OwnerUserID  string          `json:"owner_user_id,omitempty"`
	AuthScope    string          `json:"auth_scope,omitempty"`
	Code         string          `json:"code,omitempty"`
	CreatedAt    *time.Time      `json:"created_at,omitempty"`
	UpdatedAt    *time.Time      `json:"updated_at,omitempty"`
}

ChannelRecord imports an authoring channel independently of descriptive entities. Names and text preserve existing values when omitted. Scope and owner attribution do not themselves assign any new editorial grants.

type CommentRecord 

type CommentRecord struct {
	ExternalID string    `json:"external_id"`
	ID         string    `json:"id,omitempty"`
	Target     TargetRef `json:"target"`
	// Language is the language the comment is written in; empty is unknown.
	Language string `json:"language,omitempty"`
	ReplyTo  *Ref   `json:"reply_to,omitempty"`
	// UserID or, for an anonymous comment, AnonName.
	UserID    string     `json:"user_id,omitempty"`
	AnonName  string     `json:"anon_name,omitempty"`
	Body      string     `json:"body"`
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

CommentRecord is a comment on a content item (or one version), or a reply to a top-level comment on it; a reply to a reply joins its thread.

type ContentRecord 

type ContentRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Kind       string `json:"kind"`
	// State is draft, published or deleted. PublishAt schedules durable
	// publication; readers see the item only after its worker completes.
	// PublishedAt preserves a historical first-publication date.
	State       string     `json:"state"`
	PublishAt   *time.Time `json:"publish_at,omitempty"`
	PublishedAt *time.Time `json:"published_at,omitempty"`
	// DeletedAt dates a deleted item (now when omitted).
	DeletedAt *time.Time `json:"deleted_at,omitempty"`
	// Comments is open (the default), closed or hidden.
	Comments string `json:"comments,omitempty"`
	// Access defaults to public. A entity rule names the entity's id.
	Access       *access.Rule      `json:"access,omitempty"`
	Restrictions []string          `json:"restrictions,omitempty"`
	Attributes   map[string]any    `json:"attributes,omitempty"`
	Channel      *Ref              `json:"channel,omitempty"`
	Creators     []catalog.Creator `json:"creators,omitempty"`
	// Versions: a kind with hidden versions takes none or one.
	Versions  []VersionRecord `json:"versions,omitempty"`
	Code      string          `json:"code,omitempty"`
	CreatedAt *time.Time      `json:"created_at,omitempty"`
	UpdatedAt *time.Time      `json:"updated_at,omitempty"`
}

ContentRecord is a content item with its text, creators and versions. Nil text, creators and version text keep what is stored; given ones replace it. Versions not listed are left as they are. Import chapters separately through the version segments route after the content and source media exist.

type DeleteRecord added in v0.85.0 

type DeleteRecord struct {
	// Entity is content, version, entity, comment or poll.
	Entity     string `json:"entity"`
	ExternalID string `json:"external_id"`
}

DeleteRecord deletes a record the source no longer has, as its delete route would: a content item, version, entity, comment or poll, by its external id under the batch's source. One already deleted (or since erased) stays as it is, so re-sending converges; an external id never imported fails not_found.

type EdgeRecord 

type EdgeRecord struct {
	Relation string `json:"relation"`
	To       Ref    `json:"to"`
}

EdgeRecord is a relation from the entity to another.

type EntityRecord added in v0.117.0 

type EntityRecord struct {
	ExternalID string `json:"external_id"`
	// ID is kept when the external id is new (contentref.LegacyID derives a
	// stable one); generated when empty.
	ID   string `json:"id,omitempty"`
	Kind string `json:"kind"`
	Slug string `json:"slug"`
	// Internal classifications can drive domain policy but never public metadata.
	// Omission preserves an existing entity's classification; new entities are public.
	Internal *bool `json:"internal,omitempty"`
	// Scope is the entity a scoped kind's slug is unique within.
	Scope  *Ref `json:"scope,omitempty"`
	Parent *Ref `json:"parent,omitempty"`
	// State is active (the default) or deleted.
	State      string          `json:"state,omitempty"`
	Names      []entities.Name `json:"names,omitempty"`
	Text       []entities.Text `json:"text,omitempty"`
	Attributes map[string]any  `json:"attributes,omitempty"`
	// CoverImage is an image of the entity's media folder (i-{uuid});
	// CoverContent a member whose cover it shows, imported before.
	CoverImage   string `json:"cover_image,omitempty"`
	CoverContent *Ref   `json:"cover_content,omitempty"`
	// Avatar is an image of the entity's media folder.
	Avatar string `json:"avatar,omitempty"`
	// Edges replace the entity's edges to other entities.
	Edges []EdgeRecord `json:"edges,omitempty"`
	// Code keeps a content code the entity had (9 characters); one is
	// generated otherwise. A code never changes once given.
	Code      string     `json:"code,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

EntityRecord is an entity with its names and text. Nil lists keep what is stored; given ones replace it.

type ExternalID added in v0.86.0 

type ExternalID struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id"`
	Deleted    bool   `json:"deleted"`
}

ExternalID is one imported record and its id. Deleted: ContentKit holds it deleted or no longer holds it, so it needs no delete record.

type ExternalIDList added in v0.86.0 

type ExternalIDList struct {
	// Entity is content, version, entity, comment, poll or poll_option; each
	// at most once per query.
	Entity string `json:"entity"`
	// Cursor is the previous page's next_cursor.
	Cursor string `json:"cursor,omitempty"`
	// Limit defaults to 1000 and is capped at 10,000.
	Limit int `json:"limit,omitempty"`
}

ExternalIDList asks one page of an entity's external ids.

type ExternalIDPage added in v0.86.0 

type ExternalIDPage struct {
	Entity string       `json:"entity"`
	IDs    []ExternalID `json:"ids"`
	// NextCursor pages on; empty on the last page.
	NextCursor string `json:"next_cursor,omitempty"`
}

ExternalIDPage is one entity's page, in external id order.

type ExternalIDs added in v0.86.0 

type ExternalIDs struct {
	Lists []ExternalIDPage `json:"lists"`
}

ExternalIDs answers an ExternalIDsQuery: a page per list, in order.

type ExternalIDsQuery added in v0.86.0 

type ExternalIDsQuery struct {
	Source string           `json:"source"`
	Lists  []ExternalIDList `json:"lists"`
}

ExternalIDsQuery asks which external ids imports under Source hold, one cursor-paged list per entity: what a persistent sync compares with its source to find the deletes to send.

type FavoriteRecord 

type FavoriteRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

FavoriteRecord is a person's bookmark of a content item or version.

type LegacyReadingRecord added in v0.104.0 

type LegacyReadingRecord struct {
	Target      TargetRef  `json:"target"`
	UserID      string     `json:"user_id,omitempty"`
	AnonKey     string     `json:"anon_key,omitempty"`
	FirstReadAt *time.Time `json:"first_read_at,omitempty"`
	LastReadAt  time.Time  `json:"last_read_at"`
	Progress    uint32     `json:"progress"`
	ProgressMax uint32     `json:"progress_max"`
	Completed   bool       `json:"completed"`
	Resume      string     `json:"resume,omitempty"`
	Revision    uint64     `json:"revision,omitempty"`
}

LegacyReadingRecord preserves source bookmarks/progress as legacy state, never as measured attention or popularity evidence.

type MembershipRecord 

type MembershipRecord struct {
	Entity   Ref    `json:"group"`
	Version  *Ref   `json:"version,omitempty"`
	Role     string `json:"role,omitempty"`
	Position *int   `json:"position,omitempty"`
	// State is active (the default) or proposed.
	State string `json:"state,omitempty"`
}

MembershipRecord places the content item, or one version of it, in a entity.

type MembershipSet 

type MembershipSet struct {
	Content     Ref                `json:"content"`
	Kinds       []string           `json:"kinds"`
	Memberships []MembershipRecord `json:"memberships"`
}

MembershipSet replaces a content item's memberships in the listed entity kinds; memberships in other kinds stay.

type Options 

type Options struct {
	Pool     *pgxpool.Pool
	Schema   string
	Tenant   string
	Catalog  *catalog.Store
	Entities *entities.Store
	Channels *channels.Store
	Content  *content.Runtime
	// Signals is nil, or answers discovery.ErrNoSignals, without a signal
	// plane: signals are skipped.
	Signals *discovery.Service
	// Rebuild queues the jobs that rebuild what an import changed: search
	// documents, entity counts and listing counts.
	Rebuild func(ctx context.Context) error
	Logger  *slog.Logger
}

Options configures a Service over the engine's stores.

type PollOptionRecord 

type PollOptionRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	Label      string `json:"label"`
	Position   int    `json:"position"`
}

PollOptionRecord is one choice of a multiple-choice poll, ordered by Position.

type PollRecord 

type PollRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	// Kind is multiple_choice (the default and only supported kind).
	Kind     string `json:"kind,omitempty"`
	Question string `json:"question"`
	Language string `json:"language,omitempty"`
	// Active defaults to true.
	Active    *bool              `json:"active,omitempty"`
	LiveAt    *time.Time         `json:"live_at,omitempty"`
	ClosesAt  *time.Time         `json:"closes_at,omitempty"`
	DeletedAt *time.Time         `json:"deleted_at,omitempty"`
	CreatedAt *time.Time         `json:"created_at,omitempty"`
	UpdatedAt *time.Time         `json:"updated_at,omitempty"`
	Options   []PollOptionRecord `json:"options,omitempty"`
}

PollRecord is a site poll with its options, which replace the stored ones.

type ReactionRecord 

type ReactionRecord struct {
	Target    TargetRef  `json:"target"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	Value     int16      `json:"value"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
	UpdatedAt *time.Time `json:"updated_at,omitempty"`
}

ReactionRecord is a person's like (1), dislike (-1) or neutral (0) on a content item, version, comment or entity; an anonymous one is keyed by IP.

type RecordFailure 

type RecordFailure struct {
	Section    string `json:"section"`
	Index      int    `json:"index"`
	ExternalID string `json:"external_id,omitempty"`
	// Code is invalid, not_found, conflict, erased or internal.
	Code  string `json:"code"`
	Error string `json:"error"`
}

RecordFailure is one record not written. Sending it again (or the whole batch) once its cause is fixed resumes the import.

type Ref 

type Ref struct {
	ID         string `json:"id,omitempty"`
	ExternalID string `json:"external_id,omitempty"`
}

Ref names a record by its ContentKit id or by its external id under the batch's source; the entity is the field's.

type Result 

type Result struct {
	Written map[string]int `json:"written"`
	// Skipped counts records an engine without the capability leaves out:
	// signals without a signal plane.
	Skipped map[string]int  `json:"skipped,omitempty"`
	Failed  []RecordFailure `json:"failed"`
	// IDs maps each written record's external id to its id, by entity.
	IDs map[string]map[string]string `json:"ids"`
}

Result reports a batch: what each section wrote, the records that failed and the ids of the records with external ids.

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service is one tenant's bulk import.

func New 

func New(o Options) (*Service, error)

New checks the options.

func (*Service) ExternalIDs added in v0.86.0 

func (s *Service) ExternalIDs(ctx context.Context, q ExternalIDsQuery) (ExternalIDs, error)

ExternalIDs pages the external ids imports under q.Source hold, per entity.

func (*Service) Handler 

func (s *Service) Handler(a auth.Authenticator) (http.Handler, error)

Handler serves POST /v1/import and /v1/import/external-ids to holders of access.ImportWrite.

func (*Service) Import 

func (s *Service) Import(ctx context.Context, b Batch) (Result, error)

Import writes a batch, section by section. It fails as a whole only for a batch it refuses outright or when ctx ends; records that fail are in the result.

type SignalRecord 

type SignalRecord struct {
	Target TargetRef `json:"target"`
	// UserID or, signed out, an opaque AnonKey.
	UserID  string `json:"user_id,omitempty"`
	AnonKey string `json:"anon_key,omitempty"`
	// Type is click; imported raw opens are rejected.
	Type string `json:"type"`
	// EventID is the event's stable identity: re-sending it adds nothing.
	EventID    string    `json:"event_id"`
	Revision   uint64    `json:"revision,omitempty"`
	OccurredAt time.Time `json:"occurred_at"`
}

SignalRecord imports an explicit click. Existing bookmarks/progress use LegacyReadingRecord; raw opens and imported counters are not accepted.

type TargetRef 

type TargetRef struct {
	Content *Ref `json:"content,omitempty"`
	Version *Ref `json:"version,omitempty"`
	Entity  *Ref `json:"group,omitempty"`
	Channel *Ref `json:"channel,omitempty"`
	Comment *Ref `json:"comment,omitempty"`
}

TargetRef is what an interaction or alias is about: a content item, one version (its content is implied), an entity or a comment, as the section allows.

type VersionRecord 

type VersionRecord struct {
	ExternalID string `json:"external_id"`
	ID         string `json:"id,omitempty"`
	// Code is the version code (?v=) to keep, as Code is the content's: empty
	// keeps the stored one or assigns one.
	Code      string   `json:"code,omitempty"`
	Languages []string `json:"languages"`
	Traits    []string `json:"traits,omitempty"`
	Label     string   `json:"label,omitempty"`
	// State is published (the default), draft or deleted; times as ContentRecord's.
	State       string         `json:"state,omitempty"`
	PublishAt   *time.Time     `json:"publish_at,omitempty"`
	PublishedAt *time.Time     `json:"published_at,omitempty"`
	DeletedAt   *time.Time     `json:"deleted_at,omitempty"`
	Access      *access.Rule   `json:"access,omitempty"`
	Attributes  map[string]any `json:"attributes,omitempty"`
	Ordinal     *int           `json:"ordinal,omitempty"`
	// Cover selects one of the version's cover uploads (i-{uuid}): a
	// record's one legacy cover is uploaded to each of its versions. Left
	// out, the stored selection stays.
	Cover            string                         `json:"cover,omitempty"`
	Text             map[string]catalog.VersionText `json:"text,omitempty"`
	OriginalTitle    *string                        `json:"original_title,omitempty"`
	OriginalLanguage *string                        `json:"original_language,omitempty"`
	// Segments must be omitted. Use the version segments route with virtual
	// start_page markers after importing content and source media.
	Segments  []catalog.Segment `json:"segments,omitempty"`
	CreatedAt *time.Time        `json:"created_at,omitempty"`
	UpdatedAt *time.Time        `json:"updated_at,omitempty"`
}

VersionRecord is one version of a content item, ordered as listed.

type VoteRecord 

type VoteRecord struct {
	Poll      Ref        `json:"poll"`
	Option    Ref        `json:"option"`
	UserID    string     `json:"user_id,omitempty"`
	IP        string     `json:"ip,omitempty"`
	CreatedAt *time.Time `json:"created_at,omitempty"`
}

VoteRecord is one person's (or, anonymous, one IP's) choice in a poll.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.