Documentation
¶
Overview ¶
Package csvimport is Atlas's in-process worker for the CSV-to-JSON Worker Type task (ADR-0139), and the CSV parser the API's upload-validation endpoint shares with it (ADR-0084).
Like every package under connector/, it rides the standard service-task seam: a CSV task compiles to a job carrying compiler.CsvImportJobTypeIndex, and Handler picks that job up off the processor goroutine, after fsync, so parsing a file never allocates on the hot path or runs on the recovery path (I1/I4). Unlike its siblings it talks to no external system — the "connector" reads a variable and writes rows back — which is why it also serves as the parser behind the operator-facing CSV upload check.
Index ¶
- Constants
- func Formats() []string
- func Handler(store VarStore, lookup ProcessLookup) job.OutputHandler
- func KnownFormat(name string) bool
- func Parse(cfg Config, data []byte) ([]map[string]any, error)
- func ParseRows(cfg Config, data []byte) ([]map[string]any, error)
- func Render(cfg Config, rows []map[string]any) (string, error)
- type Column
- type Config
- type Job
- type ProcessLookup
- type Result
- type VarStore
Constants ¶
const ( // FormatCSV is a delimited text file — MIM's *Delimited text file* agent. FormatCSV = "csv" // FormatFixedWidth is a positional file whose columns are found by character // width — MIM's *Fixed-Width text file* agent, and how a mainframe HR extract // usually arrives. FormatFixedWidth = "fixed-width" // FormatAVP is an attribute-value pair file: "name: value" lines, one record per // blank-line-separated block — MIM's *Attribute-Value Pair text file* agent. FormatAVP = "avp" )
The text-table formats this worker reads and writes (ADR-0139, amended).
All three describe the same thing — a table of records in a text file — and differ only in how a record is delimited and how a field is found inside it. So they share the Column layout, the type coercion, and everything downstream: what a process gets back is a list of row objects whichever format the file arrived in.
They are formats of one worker rather than three workers for the same reason: unlike the SQL products (ADR-0173), nothing here can be pointed at the wrong one silently. A fixed-width layout applied to an LDIF file does not quietly produce plausible rows; it fails on the first record, loudly, at the moment the file is read.
const ( OperationRead = "read" OperationWrite = "write" )
The two directions a file task runs in.
Variables ¶
This section is empty.
Functions ¶
func Formats ¶ added in v0.3.0
func Formats() []string
Formats lists the readable/writable formats, sorted, for the messages that have to say what was expected.
func Handler ¶
func Handler(store VarStore, lookup ProcessLookup) job.OutputHandler
Handler is the in-process worker for a CSV-import service task (compiler.CsvImportJobType). It converts an uploaded CSV into a JSON `rows` collection so a batch of records is ingested entirely within the process, the file having arrived through a user-task form.
It serves two authoring shapes on the one reserved job type:
- A first-class CSV-to-JSON task (ADR-0139): the source variable, delimiter, header handling, columns, and result variable are authored on the task and compiled into a worker detail, which the worker reads from the compiled process (like the mail/rest workers). This is preferred when present.
- The ADR-0087 variable convention: with no worker detail, the worker reads `csvText` and `columnConfig` up the task's scope chain and writes `rows` + `rowCount`, so already-deployed models keep running unchanged.
A missing/empty source, an absent or malformed layout, or an unparseable CSV is a worker error: the job fails and (retries exhausted) raises an incident, rather than silently producing an empty batch.
func KnownFormat ¶ added in v0.3.0
KnownFormat reports whether name is a format this worker handles. The empty string is CSV, which is what every model authored before formats existed.
func ParseRows ¶
ParseRows parses CSV data against the predefined column layout into a list of row objects ({fieldName: value}). It is a pure, side-effect-free transform that runs in the API/side-effect phase, never on the processor hot path (ADR-0084, invariant 1/5).
Type coercion is deliberately lenient: a cell that will not coerce to its declared type is kept as its raw string, so dirty records flow through to be validated and corrected rather than being rejected at ingestion — that is the point of the feature. A *structural* mismatch (a configured header absent from the file, an unparseable CSV, an invalid layout) is an error.
Types ¶
type Column ¶
type Column struct {
Name string `json:"name"` // field name in the row object (required, unique)
Header string `json:"header,omitempty"` // CSV header to read; defaults to Name when the file has a header row
Index *int `json:"index,omitempty"` // 0-based source column, used when the file has no header row
Type string `json:"type,omitempty"` // "string" (default), "number", "integer", "boolean"
// Width is the column's character count in a fixed-width file, where a field is
// found by position rather than by a delimiter or a header. Unused by the other
// formats.
Width int `json:"width,omitempty"`
}
Column maps one CSV column into a field of the produced row object. The source column is located by Header (when the file has a header row) or by the 0-based Index (when it does not); the cell is coerced to Type. This is the "predefined column layout" a Quality Manager's upload is checked against (ADR-0084).
type Config ¶
type Config struct {
Columns []Column `json:"columns"`
Delimiter string `json:"delimiter,omitempty"` // single character; defaults to ","
HasHeader *bool `json:"hasHeader,omitempty"` // defaults to true (a header row is present)
// Format selects how records and fields are delimited; empty is CSV, which is
// what every layout authored before the other formats existed means.
Format string `json:"format,omitempty"`
}
Config is the predefined column layout an uploaded CSV is parsed against.
type Job ¶ added in v0.3.0
type Job struct {
Source string `json:"source"`
Delimiter string `json:"delimiter,omitempty"`
HasHeader bool `json:"hasHeader"`
Columns []Column `json:"columns,omitempty"`
// Format is csv (the default and what every model authored before formats
// existed), fixed-width, or avp. Operation is read (the default) or write.
Format string `json:"format,omitempty"`
Operation string `json:"operation,omitempty"`
// Result names the process variable the outcome is written to; empty means the
// default. On a read that is the rows; on a write it is the rendered file.
Result string `json:"resultVariable,omitempty"`
}
Job is a CSV-import task with everything already looked up: the text to parse and the layout to parse it against. It is what travels with a leased job.
func Resolve ¶ added in v0.3.0
func Resolve(store VarStore, cp *compiler.CompiledProcess, detail *compiler.ConnectorTaskDetail, elementInstanceKey uint64) (Job, error)
Resolve turns a compiled CSV task into a Job: the authored layout from the detail, and the source text read up the task's scope chain. It is engine work by necessity — both of its inputs live only there.
type ProcessLookup ¶
type ProcessLookup func(defKey uint64) *compiler.CompiledProcess
ProcessLookup resolves a process-definition key to its compiled process, so the worker can read a CSV worker task's authored layout from the model it belongs to (ADR-0139) — mirroring the mail/rest/DMN workers' ProcessLookup.
type Result ¶ added in v0.3.0
Result is what running a Job produces: on a read the rows as JSON, on a write the rendered file, and either way how many records were involved and the variable to put the outcome in.
func Run ¶ added in v0.3.0
Run parses a resolved job. It applies the defaults itself rather than leaving them to callers, so the in-process path and a worker cannot disagree about what an unset delimiter or result variable means.
A missing source or an unparseable file is an error, never an empty batch: a silently empty result is the failure this worker exists to avoid.
type VarStore ¶
type VarStore interface {
VariablesOfScope(scope uint64, fn func(v *model.VariableValue) error) error
GetElementInstance(key uint64) (*model.ElementInstanceValue, bool, error)
}
VarStore is the slice of the state store the CSV-import worker reads — a scope's variables and an element instance's parent scope. A narrow interface so the scope-chain walk (and its error paths) are testable with a fake (*state.Store satisfies it).
Source Files