sources

package
v0.7.10-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package sources is the input-source seam. An input source is where a run's inputs come from beyond what was uploaded: a repository, a file at a web address, a SharePoint library later. A QuickAgent asks for one with an intake question of the kind's type; the answer is fetched before the run, by the runner image's `source` role in the files container behind a proxy to the source's host alone, into /inputs/<question id>/, and decoded like any input.

Each kind is one entry in a registry and gives: how its answer is parsed and checked, the host its fetch reaches, how it uses a credential, the fetch step's argv (the fetch itself is the runner image's), its caps, its malware-check rule (the source rule, or decoded as an upload) and its receipt line. The engine runs every source through the same steps (engine/sources.go). A new kind is a new entry here and in the runner image, and a branch in Line, Document and CheckReport when its report has a shape of its own, as a url source's one file does.

A source is outside the contract: giving it is the grant, and its access token is used by the fetch step alone. The token is never part of an intake document, a receipt or an error.

Index

Constants

View Source
const (
	TokenFromFlag = "flag"
	// TokenFromBuildFlag is a knowledge build's --source-token.
	TokenFromBuildFlag = "source-token"
	TokenFromKeychain  = "keychain"
	// TokenFromSealed is the saved key, read from the keys sealed beside
	// a schedule that runs while its person is signed out.
	TokenFromSealed = "sealed"
	TokenFromPrompt = "prompt"
)

Where a token came from.

View Source
const (
	TokenFlag      = "--intake-token"
	BuildTokenFlag = "--source-token"
)

The flags that name a source's token: a run's by its question id, a knowledge build's by its host.

Variables

View Source
var HostedRepositoryCaps = Caps{Files: 10_000, Bytes: 256 << 20}

HostedRepositoryCaps are a hosted run's repository caps: 10,000 files and 256 MiB, on a run host shared by every account.

View Source
var HostedURLCaps = Caps{Files: 1, Bytes: 10 << 20}

HostedURLCaps are a hosted run's url source caps: the hosted upload cap, 10 MiB, since its file is decoded as an upload is.

View Source
var LocalCaps = Caps{Files: source.DefaultMaxFiles, Bytes: source.DefaultMaxBytes}

LocalCaps are a local repository's: 20,000 files and 1 GiB.

View Source
var URLCaps = Caps{Files: 1, Bytes: source.DefaultURLMaxBytes}

URLCaps are a local url source's: the uploads' caps, so one file within the most of one document the host reads back from a decode (256 MiB), its archive members counted against the uploads' 2,000 in the decode.

Functions

func CheckReport

func CheckReport(raw []byte, s Source, caps Caps) (source.Report, error)

CheckReport reads the fetch step's report and holds it to the source it was asked for and its caps. Everything in it was written by a process that read a host's answer, so it is held to shapes, never trusted for more than counts and words in a closed vocabulary.

func Checks

func Checks(id string, files []runnerdecode.File) string

Checks is the malware-check clause of a source's receipt line, from the decode's report of its files: how many executable or binary risky files ClamAV checked and what it made of them, and how many were left unchecked by the source rule.

func ChecksAsUpload

func ChecksAsUpload(id string, files []runnerdecode.File) string

ChecksAsUpload is the malware-check clause of a source decoded as an upload (Kind.AsUpload): every file of it, an archive's members included, was sent to the check as an upload's is; the clause counts those it checked and, apart, those it could not.

func ChecksAsUploadNamed

func ChecksAsUploadNamed(name string, files []runnerdecode.File) string

ChecksAsUploadNamed is ChecksAsUpload for a file staged under its own name rather than in a folder of the source's id (a knowledge build's): the file itself and the members of an archive it may be.

func ChecksFor

func ChecksFor(s Source, files []runnerdecode.File) string

ChecksFor is the malware-check clause for a source by its kind's rule.

func Document

func Document(s Source, rep source.Report, inputsMount string) map[string]any

Document is the source's entry in intake.json: {"repository": {"url", "ref", "commit", "path", "files"}}. It never holds the token.

func KeyAccount

func KeyAccount(host string) string

KeyAccount is the keychain account of a host's saved key.

func Line

func Line(s Source, f Fetched, checks string) string

Line is the source's receipt line: its kind, the address, the ref asked for and resolved, the commit, the files and bytes, whether a token was used (never its value), the time it took, what was left out, and the malware check of its files (checks, from the decode).

func ParseBuildTokenFlags

func ParseBuildTokenFlags(flags []string) (map[string]string, error)

ParseBuildTokenFlags reads --source-token HOST=ENV_NAME flags.

func ParseTokenFlags

func ParseTokenFlags(flags []string) (map[string]string, error)

ParseTokenFlags reads --intake-token id=ENV_NAME flags.

func ResolveBuildTokens

func ResolveBuildTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ask TokenAsker, warn func(string)) error

ResolveBuildTokens is ResolveTokens for a build's sources, whose flags name a host rather than a question: every source on that host takes the variable. A host no source is on is an error.

func ResolveTokens

func ResolveTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ask TokenAsker, warn func(string)) error

ResolveTokens fills each source's token in order: the flag's variable, the saved key, the terminal (nil: none). A flag naming a variable that is unset or empty is an error, never a quiet anonymous fetch.

func SaveToken

func SaveToken(host, token string) error

SaveToken saves a host's key in the system keychain.

func SavedToken

func SavedToken(host string) (string, error)

SavedToken is the key saved for a host, "" when there is none or the keychain is off (AGENTRY_NO_KEYRING).

func Unfetched

func Unfetched(id string, t manifest.IntakeType) error

Unfetched says why an intake type that asks for a source cannot be answered by this CLI.

Types

type Caps

type Caps struct {
	Files int
	Bytes int64
}

Caps bound one source: a source over either is refused with its count and never staged in part.

func (Caps) Within

func (c Caps) Within(files int, bytes int64) Caps

Within is the caps held to a caller's own bounds: a knowledge build's pack cap, say, which a source may not pass however large the kind allows. A bound of zero or less leaves that cap as it is.

type Fetched

type Fetched struct {
	Source source.Report
	// Took is how long the fetch step ran.
	Took time.Duration
}

Fetched is what a source's fetch step reported and the host checked.

type Kind

type Kind struct {
	// Type is the intake question type that asks for it.
	Type manifest.IntakeType
	// Noun names the kind in what a person reads.
	Noun string
	// Parse checks an answer and answers the source it names.
	Parse func(answer string) (Source, error)
	// Argv is the fetch step's argv after the runner binary: the role, the
	// source, where it writes and through which proxy, and its caps.
	Argv func(s Source, caps Caps, proxy string) []string
	// Prompt is what a terminal asks for it.
	Prompt string
	// Runner is the kind's word in the runner image's source role (its
	// --kind) and in the role's report.
	Runner string
	// Caps are a local run's caps for it; HostedCaps a hosted run's.
	Caps       Caps
	HostedCaps Caps
	// AsUpload decodes it as an upload is decoded: every file checked by
	// ClamAV, images read with OCR, and its files counted against the
	// uploads' cap. Otherwise the source rule holds: only executable
	// and binary risky files are checked, images are not read with OCR,
	// and the source has a cap of its own.
	AsUpload bool
}

Kind is one source kind on the seam.

func Lookup

func Lookup(t manifest.IntakeType) (Kind, bool)

Lookup answers the kind an intake type asks for, false for a type that asks for no source or a kind this CLI cannot fetch.

func (Kind) CapsFor

func (k Kind) CapsFor(hosted bool) Caps

CapsFor is the kind's caps for a local run or, hosted, for a run on a sandbox host shared by every account.

type Source

type Source struct {
	// ID is the intake question's id, which names the folder.
	ID   string
	Kind manifest.IntakeType
	// URL is the address, with no credentials; Ref the ref asked for,
	// empty for the default.
	URL string
	Ref string
	// Host is the address's host, lower-case: the one host the fetch step
	// reaches, and the saved key's name.
	Host string
	// Token is the access token, empty for anonymous. It goes to the
	// fetch step's stdin and nowhere else.
	Token string `json:"-"`
	// TokenFrom says where the token came from (TokenFrom*), for the
	// receipt; empty when there is none.
	TokenFrom string `json:"-"`
}

Source is one source a run fetches: a question's answer, parsed.

func (Source) Shown

func (s Source) Shown() string

Shown is the address as a person and the agent read it: a query, which can carry a signed download's credential, is shown as "?…" and never written to intake.json, a receipt or an error.

type TokenAsker

type TokenAsker interface {
	AskSecret(prompt string) (string, error)
	Confirm(prompt string, def bool) (bool, error)
}

TokenAsker asks for a token on a terminal and offers to save it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL