Documentation
¶
Overview ¶
Package sources is the input-source seam. An input source is where a run's inputs come from beyond what was uploaded: a repository, a file at a web address, a SharePoint library later. A QuickAgent asks for one with an intake question of the kind's type; the answer is fetched before the run, by the runner image's `source` role in the files container behind a proxy to the source's host alone, into /inputs/<question id>/, and decoded like any input.
Each kind is one entry in a registry and gives: how its answer is parsed and checked, the host its fetch reaches, how it uses a credential, the fetch step's argv (the fetch itself is the runner image's), its caps, its malware-check rule (the source rule, or decoded as an upload) and its receipt line. The engine runs every source through the same steps (engine/sources.go). A new kind is a new entry here and in the runner image, and a branch in Line, Document and CheckReport when its report has a shape of its own, as a url source's one file does.
A source is outside the contract: giving it is the grant, and its access token is used by the fetch step alone. The token is never part of an intake document, a receipt or an error.
Index ¶
- Constants
- Variables
- func CheckReport(raw []byte, s Source, caps Caps) (source.Report, error)
- func Checks(id string, files []runnerdecode.File) string
- func ChecksAsUpload(id string, files []runnerdecode.File) string
- func ChecksAsUploadNamed(name string, files []runnerdecode.File) string
- func ChecksFor(s Source, files []runnerdecode.File) string
- func Document(s Source, rep source.Report, inputsMount string) map[string]any
- func KeyAccount(host string) string
- func Line(s Source, f Fetched, checks string) string
- func ParseBuildTokenFlags(flags []string) (map[string]string, error)
- func ParseTokenFlags(flags []string) (map[string]string, error)
- func ResolveBuildTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ...) error
- func ResolveTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ...) error
- func SaveToken(host, token string) error
- func SavedToken(host string) (string, error)
- func Unfetched(id string, t manifest.IntakeType) error
- type Caps
- type Fetched
- type Kind
- type Source
- type TokenAsker
Constants ¶
const ( TokenFromFlag = "flag" // TokenFromBuildFlag is a knowledge build's --source-token. TokenFromBuildFlag = "source-token" TokenFromKeychain = "keychain" // TokenFromSealed is the saved key, read from the keys sealed beside // a schedule that runs while its person is signed out. TokenFromSealed = "sealed" TokenFromPrompt = "prompt" )
Where a token came from.
const ( TokenFlag = "--intake-token" BuildTokenFlag = "--source-token" )
The flags that name a source's token: a run's by its question id, a knowledge build's by its host.
Variables ¶
var HostedRepositoryCaps = Caps{Files: 10_000, Bytes: 256 << 20}
HostedRepositoryCaps are a hosted run's repository caps: 10,000 files and 256 MiB, on a run host shared by every account.
var HostedURLCaps = Caps{Files: 1, Bytes: 10 << 20}
HostedURLCaps are a hosted run's url source caps: the hosted upload cap, 10 MiB, since its file is decoded as an upload is.
var LocalCaps = Caps{Files: source.DefaultMaxFiles, Bytes: source.DefaultMaxBytes}
LocalCaps are a local repository's: 20,000 files and 1 GiB.
var URLCaps = Caps{Files: 1, Bytes: source.DefaultURLMaxBytes}
URLCaps are a local url source's: the uploads' caps, so one file within the most of one document the host reads back from a decode (256 MiB), its archive members counted against the uploads' 2,000 in the decode.
Functions ¶
func CheckReport ¶
CheckReport reads the fetch step's report and holds it to the source it was asked for and its caps. Everything in it was written by a process that read a host's answer, so it is held to shapes, never trusted for more than counts and words in a closed vocabulary.
func Checks ¶
func Checks(id string, files []runnerdecode.File) string
Checks is the malware-check clause of a source's receipt line, from the decode's report of its files: how many executable or binary risky files ClamAV checked and what it made of them, and how many were left unchecked by the source rule.
func ChecksAsUpload ¶
func ChecksAsUpload(id string, files []runnerdecode.File) string
ChecksAsUpload is the malware-check clause of a source decoded as an upload (Kind.AsUpload): every file of it, an archive's members included, was sent to the check as an upload's is; the clause counts those it checked and, apart, those it could not.
func ChecksAsUploadNamed ¶
func ChecksAsUploadNamed(name string, files []runnerdecode.File) string
ChecksAsUploadNamed is ChecksAsUpload for a file staged under its own name rather than in a folder of the source's id (a knowledge build's): the file itself and the members of an archive it may be.
func ChecksFor ¶
func ChecksFor(s Source, files []runnerdecode.File) string
ChecksFor is the malware-check clause for a source by its kind's rule.
func Document ¶
Document is the source's entry in intake.json: {"repository": {"url", "ref", "commit", "path", "files"}}. It never holds the token.
func KeyAccount ¶
KeyAccount is the keychain account of a host's saved key.
func Line ¶
Line is the source's receipt line: its kind, the address, the ref asked for and resolved, the commit, the files and bytes, whether a token was used (never its value), the time it took, what was left out, and the malware check of its files (checks, from the decode).
func ParseBuildTokenFlags ¶
ParseBuildTokenFlags reads --source-token HOST=ENV_NAME flags.
func ParseTokenFlags ¶
ParseTokenFlags reads --intake-token id=ENV_NAME flags.
func ResolveBuildTokens ¶
func ResolveBuildTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ask TokenAsker, warn func(string)) error
ResolveBuildTokens is ResolveTokens for a build's sources, whose flags name a host rather than a question: every source on that host takes the variable. A host no source is on is an error.
func ResolveTokens ¶
func ResolveTokens(srcs []*Source, flags map[string]string, getenv func(string) string, ask TokenAsker, warn func(string)) error
ResolveTokens fills each source's token in order: the flag's variable, the saved key, the terminal (nil: none). A flag naming a variable that is unset or empty is an error, never a quiet anonymous fetch.
func SavedToken ¶
SavedToken is the key saved for a host, "" when there is none or the keychain is off (AGENTRY_NO_KEYRING).
Types ¶
type Caps ¶
Caps bound one source: a source over either is refused with its count and never staged in part.
type Fetched ¶
type Fetched struct {
Source source.Report
// Took is how long the fetch step ran.
Took time.Duration
}
Fetched is what a source's fetch step reported and the host checked.
type Kind ¶
type Kind struct {
// Type is the intake question type that asks for it.
Type manifest.IntakeType
// Noun names the kind in what a person reads.
Noun string
// Parse checks an answer and answers the source it names.
Parse func(answer string) (Source, error)
// Argv is the fetch step's argv after the runner binary: the role, the
// source, where it writes and through which proxy, and its caps.
Argv func(s Source, caps Caps, proxy string) []string
// Prompt is what a terminal asks for it.
Prompt string
// Runner is the kind's word in the runner image's source role (its
// --kind) and in the role's report.
Runner string
// Caps are a local run's caps for it; HostedCaps a hosted run's.
Caps Caps
HostedCaps Caps
// AsUpload decodes it as an upload is decoded: every file checked by
// ClamAV, images read with OCR, and its files counted against the
// uploads' cap. Otherwise the source rule holds: only executable
// and binary risky files are checked, images are not read with OCR,
// and the source has a cap of its own.
AsUpload bool
}
Kind is one source kind on the seam.
type Source ¶
type Source struct {
// ID is the intake question's id, which names the folder.
ID string
Kind manifest.IntakeType
// URL is the address, with no credentials; Ref the ref asked for,
// empty for the default.
URL string
Ref string
// Host is the address's host, lower-case: the one host the fetch step
// reaches, and the saved key's name.
Host string
// Token is the access token, empty for anonymous. It goes to the
// fetch step's stdin and nowhere else.
Token string `json:"-"`
// TokenFrom says where the token came from (TokenFrom*), for the
// receipt; empty when there is none.
TokenFrom string `json:"-"`
}
Source is one source a run fetches: a question's answer, parsed.