Documentation
¶
Index ¶
- Variables
- func CreateCompositionInfo(ctx context.Context, db UploadData, nodes model.EdgeCompositionNodes, ...) (model.EdgeCompositionNodes, model.EdgeCompositionEdges, error)
- func CreateIngestTask(ctx context.Context, db UploadData, params IngestTaskParams) (model.IngestTask, error)
- func ParseAndValidatePayload(reader io.Reader, schema IngestSchema, shouldValidateGraph, readToEnd bool) (ingest.OriginalMetadata, error)
- func ReadZippedFile(zf *zip.File) ([]byte, error)
- func ValidateGraph(decoder *json.Decoder, schema IngestSchema) error
- func ValidateZipFile(reader io.Reader) error
- func WriteAndValidateFile(fileData io.Reader, location string, validationFunc FileValidator) (string, error)
- func WriteAndValidateZip(src io.Reader, dst io.Writer) (ingest.OriginalMetadata, error)
- type FileValidator
- type IngestSchema
- type IngestTaskParams
- type IngestValidator
- type UploadData
- type ValidationReport
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidJSON = errors.New("file is not valid json")
var ZipMagicBytes = []byte{0x50, 0x4b, 0x03, 0x04}
Functions ¶
func CreateCompositionInfo ¶
func CreateCompositionInfo(ctx context.Context, db UploadData, nodes model.EdgeCompositionNodes, edges model.EdgeCompositionEdges) (model.EdgeCompositionNodes, model.EdgeCompositionEdges, error)
func CreateIngestTask ¶
func CreateIngestTask(ctx context.Context, db UploadData, params IngestTaskParams) (model.IngestTask, error)
func ParseAndValidatePayload ¶
func ParseAndValidatePayload(reader io.Reader, schema IngestSchema, shouldValidateGraph, readToEnd bool) (ingest.OriginalMetadata, error)
ParseAndValidatePayload scans a JSON stream to detect and validate the metadata tag required for ingesting graph data. It ensures that either top-level "meta" and "data" tags or a "graph" tag is present. "meta"/"data" are for existing hound collections (ad and azure). The "graph" tag supports generic ingest.
If shouldValidateGraph is true, the function will also attempt to validate the presence and structure of a "graph" tag alongside the metadata.
If readToEnd is set to true, the stream will read to the end of the file (needed for TeeReader)
func ReadZippedFile ¶
ReadZippedFile - Util Function to help read zipped files
func ValidateGraph ¶
func ValidateGraph(decoder *json.Decoder, schema IngestSchema) error
ValidateGraph validates a generic ingest graph payload from a JSON stream. The input is expected to be a JSON object containing one or both of the keys "nodes" and "edges", each mapping to an array of graph elements. Each element is validated against the corresponding JSON Schema provided in the IngestSchema struct. In addition to schema validation, this function enforces constraints not expressible in JSON Schema, such as nested objects and type homogeneity in array-valued properties.
If critical errors (e.g., malformed JSON, missing brackets) or a sufficient number of validation errors are encountered, a ValidationReport is returned as an error. If no errors are found, the function returns nil.
func ValidateZipFile ¶
func WriteAndValidateFile ¶
func WriteAndValidateZip ¶
WriteAndValidateZIP implements FileValidator for ZIP ingest files.
Types ¶
type FileValidator ¶
FileValidator defines the interface for ingest file validation. It receives a source reader (src) and a destination writer (dst). Implementations are responsible for validating the input stream, while simultaneously copying it to the destination for persistence. This abstraction supports format-agnostic payloads (e.g., JSON, ZIP)
type IngestSchema ¶
type IngestSchema struct {
NodeSchema *jsonschema.Schema
EdgeSchema *jsonschema.Schema
MetaSchema *jsonschema.Schema
}
IngestSchema holds compiled JSON schemas used to validate generic-ingested graph data. It includes separate schemas for nodes and edges, which are reused across multiple ingestion requests to avoid recompiling on every request.
func LoadIngestSchema ¶
func LoadIngestSchema() (IngestSchema, error)
LoadIngestSchema constructs the JSON schema for OpenGraph ingest payloads
type IngestTaskParams ¶
type IngestTaskParams struct {
Filename string
ProvidedFileName string
FileType model.FileType
RequestID string
JobID int64
}
func SaveIngestFile ¶
func SaveIngestFile(location string, request *http.Request, validator IngestValidator) (IngestTaskParams, error)
type IngestValidator ¶
type IngestValidator struct {
IngestSchema IngestSchema
}
IngestValidator encapsulates precompiled JSON schemas used to validate graph ingest payloads, including node and edge definitions.
This struct allows schema compilation to happen once at application startup, avoiding repeated compilation during each file ingest request.
func NewIngestValidator ¶
func NewIngestValidator(schema IngestSchema) IngestValidator
func (*IngestValidator) WriteAndValidateJSON ¶
func (s *IngestValidator) WriteAndValidateJSON(src io.Reader, dst io.Writer) (ingest.OriginalMetadata, error)
WriteAndValidateJSON implements FileValidator for JSON ingest files. It streams JSON through a validator while simultaneously writing it to disk.
This method is a member of `IngestValidator` to reuse the precompiled node and edge schemas loaded during application startup.
type UploadData ¶
type UploadData interface {
// Task handlers
CreateIngestTask(ctx context.Context, task model.IngestTask) (model.IngestTask, error)
DeleteAllIngestTasks(ctx context.Context) error
CreateCompositionInfo(ctx context.Context, nodes model.EdgeCompositionNodes, edges model.EdgeCompositionEdges) (model.EdgeCompositionNodes, model.EdgeCompositionEdges, error)
// Job handlers
CreateIngestJob(ctx context.Context, job model.IngestJob) (model.IngestJob, error)
UpdateIngestJob(ctx context.Context, job model.IngestJob) error
GetIngestJob(ctx context.Context, id int64) (model.IngestJob, error)
GetAllIngestJobs(ctx context.Context, skip int, limit int, order string, filter model.SQLFilter) ([]model.IngestJob, int, error)
GetIngestJobsWithStatus(ctx context.Context, status model.JobStatus) ([]model.IngestJob, error)
DeleteAllIngestJobs(ctx context.Context) error
CancelAllIngestJobs(ctx context.Context) error
// Completed task handlers - distinctly different from task handlers
CreateCompletedTask(ctx context.Context, task model.CompletedTask) (model.CompletedTask, error)
GetCompletedTasks(ctx context.Context, ingestJobId int64) ([]model.CompletedTask, error)
}
The UploadData interface is designed to manage the lifecycle of uploading and creating tasks for graphify
type ValidationReport ¶
type ValidationReport struct {
CriticalErrors []validationError // things like json syntax errors where the document is un-parseable
ValidationErrors []validationError // nodes and edges that dont conform to the spec
}
func (ValidationReport) BuildAPIError ¶
func (s ValidationReport) BuildAPIError() []string
func (ValidationReport) Error ¶
func (s ValidationReport) Error() string
Source Files
Directories