Documentation
¶
Overview ¶
Package ingest pulls knowledge from where a team already keeps it (GitHub, Slack, ...) into THEIR vault, on THEIR hardware - the sovereign version of what cloud search tools do. Each imported item becomes a markdown note with provenance frontmatter (source=import:<connector>, source_url, imported_at, author), written to a deterministic path so a re-pull upserts instead of duplicating. The more sources flow in, the higher the switching cost - this is the data-gravity moat.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RenderDoc ¶
RenderDoc renders one imported Doc to its vault-relative path and provenance- stamped markdown, WITHOUT touching disk. The hub uses this to commit imports through its git repo (a Change) instead of writing files; Run uses it for the local CLI path. The path is deterministic per (connector, ExternalID) so a re-pull upserts the same note.
Types ¶
type Config ¶
type Config struct {
Connectors []ConnectorConfig `json:"connectors"`
}
Config drives `mesh ingest all`: a list of connector instances to pull on each run. It lives at <vault>/.mesh/ingest.json and holds NO secrets - tokens are read from env per connector type, keeping config safe to commit/share.
Example:
{"connectors":[
{"type":"github","owner":"bright-interaction","repo":"automations"},
{"type":"slack","channel":"C0123456"},
{"type":"linear"},
{"type":"jira","site":"https://acme.atlassian.net","email":"me@acme.com"},
{"type":"notion"}
]}
func LoadConfig ¶
LoadConfig reads .mesh/ingest.json (absent = empty config, not an error).
type Connector ¶
type Connector interface {
Name() string
Key() string
Pull(ctx context.Context, since time.Time) (docs []Doc, truncated bool, err error)
}
Connector pulls docs from one external source since a timestamp (zero = all). Key is a stable per-instance id (e.g. "github:owner/repo") used to remember the last successful pull for incremental sync.
Pull MUST paginate the source to exhaustion for the window. It returns truncated= true only when it could not (it hit maxIngestPages with more data still upstream); in that case the caller must NOT advance the high-water mark, so the un-pulled tail is re-fetched next run instead of being silently skipped forever.
type ConnectorConfig ¶
type ConnectorConfig struct {
Type string `json:"type"` // github|slack|linear|jira|notion
Owner string `json:"owner,omitempty"` // github
Repo string `json:"repo,omitempty"` // github
Channel string `json:"channel,omitempty"` // slack
Site string `json:"site,omitempty"` // jira
Email string `json:"email,omitempty"` // jira
JQL string `json:"jql,omitempty"` // jira (optional)
}
ConnectorConfig is one instance. Fields used depend on type.
func (ConnectorConfig) Build ¶
func (cc ConnectorConfig) Build() (Connector, error)
Build turns a ConnectorConfig into a live Connector, reading the token from the per-type env var (never from config). Returns an error for an unknown type or a missing required field.
func (ConnectorConfig) InstanceKey ¶
func (cc ConnectorConfig) InstanceKey() string
InstanceKey is the stable per-instance id, matching the built connector's Key(). It needs no token, so callers (e.g. the hub Integrations UI) can look up a connector's saved high-water mark / last-run without constructing the connector.
type Doc ¶
type Doc struct {
ExternalID string // stable per-source id (drives the deterministic filename)
Title string
Body string
Author string
SourceURL string
CreatedAt string // YYYY-MM-DD
}
Doc is one upstream item to import.
type GitHub ¶
type GitHub struct {
Owner string
Repo string
Token string
MaxPages int // page cap (per_page=100); 0 -> 5
Client *http.Client
}
GitHub imports issues + pull requests (the issues endpoint returns both) for one repo into the vault. Auth is a token (PAT or fine-grained) from the caller; a preset OAuth app is a follow-on. Pure stdlib, no SDK.
type Jira ¶
type Jira struct {
Site string // https://your-domain.atlassian.net
Email string
Token string
JQL string // base filter; default "order by created DESC"
Max int // page size; 0 -> 100
Client *http.Client
}
Jira imports issues via the Jira Cloud REST v3 enhanced search endpoint (GET /rest/api/3/search/jql; the old /search was removed in late 2025). Auth is HTTP Basic with email + API token.
type Linear ¶
Linear imports issues via the GraphQL API. Personal API keys go in the Authorization header RAW (no "Bearer " prefix); OAuth tokens would use Bearer.
type Notion ¶
type Notion struct {
Token string
Version string // Notion-Version header; default below
Limit int // page_size; 0 -> 100
Client *http.Client
}
Notion imports pages via the search endpoint. Auth is a Bearer integration token; the Notion-Version header is required. v1 indexes page title + url + timestamps (block content needs per-page calls; a follow-on).
type Opts ¶
type Opts struct {
Full bool // ignore stored high-water mark; pull everything
Since time.Time // explicit override (wins over stored state when non-zero)
}
Opts controls an incremental run.
type Result ¶
type Result struct {
Connector string `json:"connector"`
Pulled int `json:"pulled"`
Written int `json:"written"`
Folder string `json:"folder"`
Truncated bool `json:"truncated"` // hit the page cap; mark not advanced, more to pull
}
Result reports what a run wrote.
func Run ¶
Run pulls from c and upserts each doc as a provenance-stamped note under imported/<connector>/ in vaultRoot. Idempotent: a re-pull overwrites the same deterministic file, so source_url dedupe is automatic.
func RunIncremental ¶
RunIncremental pulls only what changed since the connector's last successful run (a high-water mark persisted in <vault>/.mesh/ingest-state.json), then advances the mark. --full or an explicit Since override the stored mark. The mark is stamped from BEFORE the pull, so anything that lands mid-pull is caught next time.
Source Files