GSEER

GSEER

Internet Asset Intelligence & Search Engine. Discover. Identify. Search.

GSEER is an open-source, self-hosted internet-asset intelligence and search engine. You point it at identifiers you care about, its passive collectors gather public facts about them (DNS records, ASN ownership), and you search everything it has indexed with a simple query language — through a CLI, a REST API, and a built-in dashboard.

Everything runs on your machine: your database, your data, your rules.

Features

• Search (GQL) — one query language, three surfaces (CLI, REST API, dashboard), all backed by a single SearchService. Search is always read-only: a query can never trigger collection or mutate data.
• Faceted search — port / protocol / technology / service / country / source / freshness aggregates over the same filter (gseer search --facets).
• Export — streamed JSON / CSV / NDJSON with cursor-stable pagination and RFC 4180 escaping (gseer export).
• Passive collection — explicit, operator-driven collectors with a terms-review gate, checkpoints, dead-letter bookkeeping, and re-drive. Nothing ever runs uninvoked.
• Suppression policy — block IPs/CIDRs/domains from ever entering your index, retroactively if needed (gseer suppress).
• Fingerprint rules — a lint-gated, versioned rule collection that classifies technology evidence an observation already carries into products with confidence (gseer rules).
• Signed updates — gseer update verifies an Ed25519-signed manifest against a key embedded in the binary before anything is downloaded or replaced.
• Full Docker stack — one command writes a db + server compose file with healthchecks, a persistent volume, and a generated API token.

Requirements

v0.2.0 is a source-only self-hosted release. The artifact is the source archive GSEER-v0.2.0-source.zip together with its SHA-256 checksum sidecar GSEER-v0.2.0-source.zip.sha256. This source archive is the only downloadable release artifact for v0.2.0. There are no prebuilt platform binaries, no binary packages, and no published Docker image for this release. Build from source (below), or build the Docker image locally via the Dockerfile. Docker is the easiest way to satisfy the PostgreSQL dependency.

v0.2.1 (compatible increment) adds the GQL compatibility layer (*=, free text, field aliases), personal mode on SQLite — a single-file database with no PostgreSQL dependency — and release engineering (validated release config, go install support, draft-gated releases). See docs/releases/RELEASE_NOTES_v0.2.1.md. PostgreSQL remains the primary engine for the self-hosted server profile.

Quick Start

Copy-paste, in order. Total time: a few minutes.

# 1. Build the binary (requires Go 1.26+)
git clone https://github.com/Lutfifakee-Project/GSEER.git
cd GSEER
make build VERSION=v0.2.1
./bin/gseer version
#   → gseer v0.2.1
#   (or, without cloning:
#    go install github.com/Lutfifakee-Project/GSEER/cmd/gseer@v0.2.1)
#   (optional: sudo cp bin/gseer /usr/local/bin/ so `gseer` is on PATH)

# 2. Write the compose file + a .env with a random database password
gseer setup --self-hosted --with-compose

# 3. Start PostgreSQL (healthcheck-gated)
docker compose up -d db
docker compose ps db          # wait until it shows "healthy"

# 4. Hardened setup — bootstrap → migrate → grants → audit, one command
gseer setup --self-hosted --db-url "$(grep '^GSEER_DSN=' .env | cut -d= -f2-)"
#   → [ ok ] migrations: 8 applied, schema at head v17
#   → [ ok ] privilege audit: PASSED (fail-closed gate, OD-5)
#   → prints GSEER_DB_RUNTIME_DSN (the gseer_app credential) EXACTLY ONCE

# 5. Make this shell self-hosted (setup does not rewrite your shell)
export GSEER_PROFILE=self-hosted
export GSEER_DB_DSN="$(grep '^GSEER_DSN=' .env | cut -d= -f2-)"
export GSEER_DB_RUNTIME_DSN='postgres://gseer_app:<from step 4>@127.0.0.1:5432/gseer?sslmode=disable'
#   Without this, the CLI defaults to the client profile and doctor
#   fails with E_CONFIG_REQUIRED (client.endpoint missing).  The server
#   additionally REQUIRES the runtime credential — it never serves
#   traffic under the operator credential (docs/DEPLOYMENT.md).

# 6. Verify the installation
gseer doctor
#   → 6 PASS, 0 FAIL.  (The index is reported as EMPTY — that is honest.)

# 7. Your first search (a fresh install has an empty index)
gseer search 'port==443'
#   → The index is empty — no assets have been ingested yet.

# 8. (Optional) load the labeled SYNTHETIC evaluation corpus so you can
#    try search/export/facets without collecting anything
gseer setup --load-fixtures          # asks for a y/yes confirmation
gseer search 'port==443 && title CONTAINS "login"'
#   Every surface labels this data: SYNTHETIC EVALUATION DATA.

# 9. Start the dashboard + API (runs the privilege audit first, in the foreground)
gseer server start
#   Dashboard:  http://127.0.0.1:8443/
#   API health: http://127.0.0.1:8443/api/v1/health

Prefer everything in Docker? gseer setup --with-compose --full writes a docker-compose.full.yml (db + server, generated API token, database not exposed to the host) — then docker compose -f docker-compose.full.yml up -d --build. See docs/DOCKER.md.

Stop / restart the service:

# server start runs in the foreground:
#   stop with Ctrl-C in its terminal, or from another terminal:
gseer server stop
gseer server status
gseer server start          # start it again
docker compose stop db      # stop PostgreSQL;  docker compose start db

First searches (GQL)

GQL is the GSEER Query Language. = is sugar for ==; strings are double-quoted; numbers are bare integers.

gseer search 'port=443'                            # `=` is sugar for `==`
gseer search 'title="phpMyAdmin"'                  # strings are double-quoted
gseer search 'server="nginx" && port=443'          # boolean combine
gseer search 'ip IN_CIDR "203.0.113.0/24"'         # CIDR containment
gseer search 'domain IN_DOMAIN "example.net"'      # apex + subdomains
gseer search 'ANY(host){ http.status == 200 }'     # host has ANY service with 200
gseer search 'port==443' --facets                  # facet aggregates over the same rows
gseer search 'stale == "false"' --facets --facet port,protocol,country --max-values 5
gseer search 'port==443' --mode asset              # one row per asset
gseer search 'port==443' --explain                 # print the compiled query plan
gseer search 'port==443' --validate                # validate only, no DB contact
gseer search '"wordpress"'                         # bare-keyword free text (stored text surfaces)
gseer search 'ip*="203.0.113.*"'                   # wildcard fuzzy match (* = any run, ? = one)
gseer search 'after="2026-01-01" && status_code=200'  # compat aliases: time filter + status

The compatibility layer adds C-style operators (&&, ||, !), wildcard fuzzy matching (*=), common field aliases (host, org, base_protocol, status_code, cert.subject.cn, cert.issuer.cn), directional time filters (after/before), derived booleans (is_domain, is_ipv6, cert.is_expired), and empty-value semantics (field="" = missing-or-empty). The full mapping — including what is deliberately NOT supported and why (Search ≠ Scan) — is in docs/SEARCH-SYNTAX.md.

The full field registry (every field and its operators) is in gseer search --help. Export the results:

gseer export 'stale == "false"' --format csv --out results.csv
gseer export 'port == 443' --format ndjson --out -   # stdout; stats go to stderr

CLI Command Reference

Run gseer --help for the live list; gseer <command> --help documents every flag. Verified against v0.2.0:

Planned but not in this release: gseer scan, gseer asset. Invoking them prints a pointer and exits with code 64 — no scanning exists in GSEER.

Dashboard

gseer server start serves a server-rendered dashboard at http://127.0.0.1:8443/ (loopback by default). It is read-only (GET-only, no cookies, no login) and runs the same SearchService as the CLI: the search box accepts GQL, results show the same rows, and the facet panel mirrors --facets. The dashboard never triggers collection.

REST API

The API lives under /api/v1 on the same port. On the default loopback bind no token is required; a non-loopback bind requires one (GSEER_TOKEN, enforced by the config layer).

curl 'http://127.0.0.1:8443/api/v1/search?q=port%3D%3D443'
curl -H "Authorization: Bearer $GSEER_TOKEN" 'http://127.0.0.1:8443/api/v1/search?q=port%3D%3D443'

Data Collection

Nothing collects uninvoked. You register a source, enable it, and run it — one explicit command each. Targets are identifiers you named or assets already in your index; there are no third-party target lists and no enumeration.

# Register a passive DNS source (targets = domains you choose)
gseer source add my-dns --type dns --seed-mode config \
    --seeds example.net,example.org --enable --yes

# Run it once (explicit; no daemon, no scheduler)
gseer collect run --source my-dns

# What landed (passive facts are asset-level: a DNS answer proves the
# name was published — it never invents ports, banners or HTTP titles)
gseer search 'domain IN_DOMAIN "example.net"'

# Bookkeeping and recovery
gseer collect status                       # jobs, checkpoints, dead letters
gseer collect redrive --all                # re-ingest dead-lettered records
gseer collect redrive --source my-dns --discard

# Keep classes of records out of your index entirely
gseer suppress add cidr 10.0.0.0/8 --reason "RFC1918"
gseer suppress add domain_suffix internal.example --reason "our own space"
gseer suppress list

Connector status in v0.1.0:

Disabled connectors are enforced in code: enabling or running them refuses with E_SOURCE_TERMS. Verdicts and the review register live in docs/SOURCES.md.

Configuration

Precedence: CLI flags > environment variables > YAML config file > defaults. The config file lives at ~/.gseer/config.yaml (override with --config or GSEER_CONFIG); see .env.example for the full annotated set.

gseer setup --with-compose generates a .env (mode 0600) with a random database password — never commit a real .env.

Troubleshooting

Security, Limitations, and License

Security posture

• Search ≠ Scan — no code path in this repository port-scans, banner-grabs, or enumerates the internet. A query can only read. Planned active observation (gseer scan) does not exist in this release; invoking it exits 64.
• Nothing runs uninvoked — no daemon, no scheduler. Every collection run is an explicit operator command, gated by the terms register.
• Least-privilege runtime, fail-closed — the server serves traffic ONLY as the least-privilege gseer_app credential, and the fail-closed effective-privilege audit refuses startup (exit 5) unless the deployment's roles/grants/ownership are exactly the approved matrix. There is no bypass, flag, or override (docs/DEPLOYMENT.md).
• Suppression is enforced twice — in every ingestion path AND by a database trigger that refuses suppressed writes if any path forgets; after migration 0016 all suppression writes go through the gseer_policy SECURITY DEFINER functions, never direct DML.
• Loopback by default — the DB binds 127.0.0.1 only (compose full stack: no host port at all); the server binds loopback and a non-loopback bind requires a token; security headers on every response; API tokens are hashed at rest.
• Updates are verified before anything is downloaded — Ed25519 signature against the key embedded in the binary, then version gate, platform match, SHA-256, atomic replace with rollback.

Limitations (honest)

• Self-hosted index only. A fresh install has an empty database; you search exactly what your sources collected. There is no global, hosted-engine-scale index — GSEER is the engine, not a hosted service.
• Passive facts are asset-level. A DNS answer or a certificate-log entry proves an identifier was published; it never invents a port, a banner, or an HTTP title. Service-level rows appear only from evidence that actually carries them; DNS-discovered IPs do not mirror the resolving name into the asset's hostname/domain columns (the name is on the observation's evidence — docs/DEPLOYMENT.md §7).
• RDAP and crt.sh/CT are disabled until their providers' terms are reviewed and approved. The enforcement is in code, not just docs.
• Fingerprint coverage is deliberately conservative — the seed set classifies only well-documented header/banner evidence; CMS/framework identification extends via contributions (docs/RULES.md).
• No exploitation signatures. The rule linter rejects anything shaped like an exploit or credential payload.

License

GNU Affero General Public License v3.0 — see LICENSE. Contributions are welcome: fingerprint rules (docs/RULES.md) are the easiest starting point; every Go file carries the AGPL header and CI enforces it.

Documentation

• docs/QUICKSTART.md — the canonical fresh-install workflow
• docs/INSTALL.md — install methods, checksums, build from source
• docs/DEPLOYMENT.md — credential model, hardened startup, the fail-closed privilege audit
• docs/DOCKER.md — full stack: startup, upgrade, backup/restore, recovery
• docs/UPDATE.md — the signed-update chain, the v0.1.0 → v0.2.0 upgrade path, key ceremony, recovery
• docs/SOURCES.md — passive data-source review register (terms verdicts)
• docs/RULES.md — fingerprint rule format, confidence tiers, contributions
• docs/ROADMAP.md — release register (what is live per release)
• CHANGELOG.md — per-release changes
• docs/releases/ — per-release release notes (v0.1.0, v0.2.0, v0.2.1)