ami

ami

ami (網, "net") re-fetches every URL in a seed (a list of URLs) as fast as one machine sustains, then packs the results into rotated, zstd-compressed Parquet files. WARC output is still available with --format warc. The seed can be a text file, newline JSON, a Parquet column, a sitemap, or stdin. They all feed the same fetch engine.

Install • Quick start • Seed formats • Output • Flags • Build

A crawler is two jobs glued together: deciding which URLs to fetch, and fetching them. ami does only the second. You bring a list of URLs (a frontier someone else produced, a sitemap, a column lifted out of a dataset) and ami re-fetches every one, archives the responses in a standard format, and indexes them so you can find any single response again.

Full docs and guides live at ami.tamnd.com.

Install

# Go
go install github.com/tamnd/ami/cmd/ami@latest

# Homebrew (macOS)
brew install tamnd/tap/ami

# Scoop (Windows)
scoop bucket add tamnd https://github.com/tamnd/scoop-bucket
scoop install ami

On Linux, a signed apt and dnf repository tracks every release:

# Debian, Ubuntu
curl -fsSL https://tamnd.github.io/linux-repo/gpg.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/tamnd.gpg
echo "deb [signed-by=/usr/share/keyrings/tamnd.gpg] https://tamnd.github.io/linux-repo/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/tamnd.list
sudo apt update && sudo apt install ami

# Fedora, RHEL
sudo dnf config-manager --add-repo https://tamnd.github.io/linux-repo/dnf/tamnd.repo
sudo dnf install ami

Prefer a prebuilt binary? Grab an archive, a .deb/.rpm/.apk, or a checksum from releases. Or use the container image:

docker run -v "$PWD/out:/out" ghcr.io/tamnd/ami crawl --from lines /out/urls.txt

ami is a single pure-Go binary with no runtime dependency beyond CA roots.

Shell completion ships in the box: ami completion bash|zsh|fish|powershell.

Quick start

# A list of URLs, one per line
cat > urls.txt <<'EOF'
https://example.com/
https://example.com/about
https://www.iana.org/help/example-domains
EOF

# Crawl it: writes rotated captures-NNNNN.parquet under ami-out/
ami crawl urls.txt

# Look at what landed, no Parquet tool needed
ami inspect ami-out/

captures: 3 rows in ami-out/captures-00000.parquet

STATUS  BYTES  HOST                 URL
200     1256   example.com          https://example.com/
200     1256   example.com          https://example.com/about
200     9417   www.iana.org         https://www.iana.org/help/example-domains

The seed does not have to be a file. Pipe URLs in, or point ami at a sitemap:

printf 'https://example.com/\n' | ami crawl --from lines -
ami crawl https://www.example.com/sitemap.xml

Seed formats

A seed is just a list of URLs. ami reads four formats, inferred from the path or set with --from:

With JSONL, any field other than url rides along as per-capture metadata and lands in the index's meta_json column.

Output

A crawl writes a series of rotated Parquet files under --out (default ami-out):

ami-out/
├── captures-00000.parquet   # rows + inline bodies, zstd, rolled at --capture-size
├── captures-00001.parquet
└── captures-00002.parquet

Each file is finalized with its own footer, so it is independently readable and can be offloaded and deleted mid-run while the crawl keeps going. The bodies live inline in a body column, compressed columnar with zstd. Because zstd packs thousands of similar pages together, this is several times smaller on disk than per-record gzip WARC, and the crawl is network-bound so the heavier compression is effectively free. Each row carries these columns:

The index reads in DuckDB, pandas, or ami inspect. A prior run's capture files can be fed straight back as a recrawl seed (--from parquet): ami reads url/digest/etag/last_modified and issues conditional requests, while the heavy body/header columns are ignored.

With --format warc, ami instead writes classic WARC/1.1 files (ami-NNNNN.warc.gz, one gzip member per record) plus a metadata-only Parquet index whose warc_file/warc_offset/warc_length columns point at each record. Use it for archival fidelity and interop with the web-archiving ecosystem.

Flags

ami crawl <seed> takes:

ami crawl --help has the canonical list. ami inspect <captures.parquet> takes -n/--limit for the number of sample rows.

Building from source

git clone https://github.com/tamnd/ami
cd ami
make build          # -> bin/ami (pure Go)
make test           # full suite
make test-short     # skip the tests that hit the network

The repo is split by concern:

cmd/ami/   thin main: hands off to cli.Execute
cli/       the cobra command tree and flag wiring
seed/      the seed readers: lines, jsonl, parquet, sitemap
fetch/     the concurrent fetch engine: workers, transport, header profiles
run/       the crawl loop that ties seed, fetch, and pack together
pack/      WARC writer and the Parquet capture index
config/    the tunables for one run
metrics/   the live progress snapshot
urlx/      URL helpers
docs/      the tago documentation site

Releasing

Push a version tag and GitHub Actions runs GoReleaser, which builds the archives, the .deb/.rpm/.apk packages, a multi-arch GHCR image, checksums, SBOMs, and a cosign signature:

git tag v0.1.0
git push --tags

The image tag carries no v prefix (ghcr.io/tamnd/ami:0.1.0). The Homebrew and Scoop steps self-disable until their tokens exist, so the first release works with no extra secrets.

License

MIT. See LICENSE.