valet

valet

Valet is a validation gate for data files before they reach your warehouse or lakehouse. It checks CSV, Parquet, JSON and NDJSON files on local disk or S3-compatible storage against a YAML or JSON pipeline, and its exit code and JSON report decide whether a file moves on.

Valet is designed for short-lived batch jobs with an explicit managed-memory budget. That budget is not a hard process-RSS limit.

The concept of Valet is that on a server memory usage is more important than runtime. Since it is designed for batch data validation memory constraints often play a bigger role.

Valet outputs structured logs, structured json reports and writes data to s3 compatible storage.

Valet suits any workload that can batch ingest and run cli commands, e.g. Apache Airflow or similar.

You can pipe the output of Valet reports into any ingestion tool, but piping it into jq is a good use case to quickly query multiple valet reports.

Performance

Valet trades speed for bounded memory: it reads and checks every row itself, so its memory is capped by the managed budget while pandas holds the whole file.

Both tools reported the same failure counts. Clean, generated data on an Intel Core Ultra X7 358H (16 threads), warm page cache; medians of three fresh processes. Versions: GX 1.24.1/pandas 3.1.0, Python 3.12. Reproduce with benchmarks/competitors; method and dirty-input behaviour are in the performance guide.

Where does it fit?

Valet checks a file before anything loads it, while the original records still exist:

• Loading into staging. Validate each file drop and load it only on exit 0. With an output destination (-o), rejected rows keep their original records, so the producer can see exactly what to fix.
• Accepting large uploads. A service that accepts CSV, Parquet or JSON uploads can run Valet on the stored upload (local or S3) and accept or refuse it from the exit code and report. Memory is set by the managed budget, not by the size of the upload.
• Pipeline and CI gates. Any scheduler that runs commands, such as Airflow or a CI job, can block the next step when error-level checks fail.

Once data is in the warehouse, warehouse-side tools such as Soda or GX can monitor it. Valet complements them; it does not replace them.

What can it do?

• Read CSV, Parquet, JSON and NDJSON from local files or S3-compatible storage.
• Check schemas, missing values, ranges, duplicate keys, references and column relationships.
• Return a JSON report, or optionally write CSV/Parquet row outputs and detailed issues.
• Treat values that fail conversion (for example abc in a decimal column) as row results: the row fails, the cell is counted, and with an output destination the original record is written to rejects.
• Compare integers and decimals exactly, without rounding through floats.

Structurally malformed input (a CSV row with the wrong number of fields, broken quoting or invalid UTF-8) stops the run with exit 2 and an error naming the file, the line and how to fix it. With input.on_malformed: reject, CSV rows with the wrong number of fields become rejects instead and the rest of the file is still checked.

Why use it?

Catch bad data before loading it downstream. Keep your checks in YAML or JSON and use Valet's exit code to decide whether your pipeline continues—without writing a validation script for each input or deploying another service.

How does it compare?

Valet runs with no Python dependency.

GX also reads files directly (GX). Choose Valet for a standalone file-validation gate, not a hosted platform.

Measured deployment footprint

Valet is one self-contained executable. It does not need Python or any other runtime; GX Core needs a Python environment.

Measured versions: GX 1.24.0/pandas 3.0.6 on Python 3.12.15; Valet built from this repository with the release flags (-trimpath -ldflags="-s -w").

Install

Download a binary, or use Go 1.27.3+:

go install github.com/richardrh/valet/cmd/valet@latest

Run

jq is not required but the below example shows how a user can filter or format the output by piping into other cli tools.

valet validate -p pipeline.yml -i input.parquet > report.json
jq '{validation, total, pass, warn, error}' report.json

Exit 0: passed (warnings allowed). Exit 1: failed checks. Exit 2: execution failed. Use Valet's exit code—not jq's—to gate the next step.

Editors can autocomplete and check pipelines against the published JSON Schema; AI agents can load all docs from llms-full.txt. See editor support and AI agents.

Docs and examples · MIT License