dbs

package
v0.7.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 24, 2025 License: BSD-3-Clause, GPL-3.0 Imports: 29 Imported by: 0

README

DBs

Usage Options

Users have three options for obtaining netcap databases:

  1. Download from server (Recommended): net util -download-dbs
  2. Generate locally: net util -generate-dbs
  3. Set up your own database server: See Database Server section below

Database Server

Netcap now includes a built-in HTTP server that automatically rebuilds and serves databases with nightly updates.

Starting the Server
# Start the database server
net util -serve-dbs

# Specify custom address
net util -serve-dbs -serve-addr :9090

# With verbose logging
net util -serve-dbs -verbose

See docker/dbs-server/README.md for complete documentation.

# Quick start with docker-compose
cd docker/dbs-server
docker-compose up -d
Downloading Databases
# Download from default URL (dbs.netcap.io)
net util -download-dbs

# Download from custom server
net util -download-dbs -dbs-url http://your-server:8080

# Using environment variable
export NETCAP_DBS_URL=http://your-server:8080
net util -download-dbs

# Force re-download
net util -download-dbs -force
API Endpoints
  • GET /health - Health check
  • GET /dbs/latest - Latest version metadata (JSON)
  • GET /dbs/list - List all available versions (JSON)
  • GET /dbs/latest.tar.gz - Download latest database tarball
  • GET /dbs/YYYY-MM-DD.tar.gz - Download specific version
Database Storage

The database server stores files in the following structure:

netcap-dbs-server/          # Root directory (configurable via NC_CONFIG_ROOT)
├── dbs/                    # Database storage directory
│   ├── 2024-01-15.tar.gz  # Versioned database tarball
│   ├── 2024-01-15.json    # Metadata for version
│   ├── latest.tar.gz      # Symlink/copy of latest version
│   └── latest.json        # Symlink/copy of latest metadata
└── build/                  # Temporary build directory

Configuration:

  • Set NC_CONFIG_ROOT environment variable to change the root directory
  • Default: netcap-dbs-server (relative to current working directory)
  • Docker default: /data/netcap-dbs-server

Using Pre-existing Databases:

The server can use pre-existing databases instead of rebuilding on startup. Simply mount or copy database files into the dbs/ directory before starting the server. The server will:

  1. Detect existing database tarballs (YYYY-MM-DD.tar.gz format)
  2. Use the most recent version as the initial revision
  3. Create latest symlinks automatically
  4. Skip initial rebuild and start serving immediately
  5. Continue with scheduled nightly rebuilds

For detailed instructions on mounting databases with Docker, see docker/dbs-server/README.md.

TODOs

  • integrate https://github.com/malware-traffic/indicators

  • initJa3Resolver: index ja3 json dbs in bleve and bundle with dbs

  • merge PR to add fault tolerance to build process

  • integrate Ja4+ and deprecate Ja3

  • add generic interface for netcap dbs, so that custom data or feeds can be easily integrated

  • deprecate netcap-dbs repo

Additiontal Data

NETCAP DBs

This is a collection of various open sourced databases with information that netcap uses for audit record enrichment and correlation.

Some data sources are used in original form, some are preprocessed.

Index

TODOs

Installation

To clone this repo you need to install the LFS git plugin to handle large files.

Apt/deb: sudo apt-get install git-lfs
Yum/rpm: sudo yum install git-lfs
MacOS: brew install git-lfs
Windows: ???

If you want to contribute to the repository, you will need to install the lfs and license checker hooks with:

./install-hooks.sh

Data Sources

The following data sources are included:

Wappalyzer Technologies Database

Provides common attributes of web frameworks for identification.

Source: https://github.com/AliasIO/wappalyzer/blob/master/src/technologies.json

License: MIT

Fingerbank Open Sourced DHCP Fingerprints

Fingerprinted DHCP devices will be enriched with information from: https://raw.githubusercontent.com/karottc/fingerbank/master/upstream/startup/fingerprints.csv

It can be used to get a small fraction of the Fingerbank database for offline lookups, however beware these are likely outdated.

It is also possible to authenticate to the Fingerbank API via API key for more accurate lookups.

License: Commercial

Domain Whitelist (Alexa Top 1 million)

Commonly seen domains on the web, mostly from legitimate companies not known to distribute malicious software.

Be aware that some malicious domains made it into this list in the past, but this list is still useful to filter out likely harmless traffic on big datasets.

Source: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

License: MIT

MaxMind GeoLite2 CC Databases

For retrieving geographic City and ASN information about an IP address.

This repository contains the last version of the database that was distributed under the Creative Commons license (from the 27th of December 2019, backed up the by the web archive).

For obtaining the latest version, you have to sign up at maxmind and agree to their terms of service:

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases

To ensure the latest version is not accidentally pushed into the repository pre-commit and post-commit git hooks are used, essentially hot swapping the versions in the repository root with the CC licensed ones on every commit.

License: CC / Commercial

HASSHDB from AdelKa

Various SSH fingerprints, used to enrich SSH audit records.

Source: https://raw.githubusercontent.com/0x4D31/hassh-utils/master/hasshdb

License: BSD3

Ja3 associated Client and Server Fingerprints

Associated Ja3 client and server fingerprints for a handful of OS and browser variants.

Recorded in our lab environment during our research project for the Offensive Technologies course.

Used to increase accuracy for software identification.

Filename: ja_3_3s.json

License: MIT

Ja3 Fingerprints and UserAgents from Ja3er.com

TLS client and server hashes and associated user agents for threat hunting, from https://ja3er.com.

Source Hashes: https://ja3er.com/getAllHashesJson

Source UserAgents: https://ja3er.com/getAllUasJson

License: None provided

Trisul Ja3 Fingerprints

https://github.com/trisulnsm/trisul-scripts/blob/master/lua/frontend_scripts/reassembly/ja3/prints/ja3fingerprint.json

TODO: integrate new repo https://github.com/trisulnsm/ja3prints

License: None provided

Macaddress.io Database

Mac OUI to Vendor Names and registered addresses.

Source: https://macaddress.io/database-download

License: https://macaddress.io/terms-of-service

Nmap Service Probes

https://svn.nmap.org/nmap/nmap-service-probes

License: NPSL (https://nmap.org/npsl)

User Agent Parser Regexes

Regular expressions to identify the software behind a useragent more accurately.

Used to create additional software audit records based on HTTP user agent observations.

Source: https://raw.githubusercontent.com/tobie/ua-parser/master/regexes.yaml

License: Apache 2

IANA Service Names to Port Numbers

Ports mapped to services for TCP and UDP, used to enrich the service audit records.

Source: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv

NVD vulnerabilities indexed in a BleveDB

Used to lookup identified software products and search for known vulnerabilities.

Indexed in a bleve database.

Source: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED

License:

The entire NVD database can be downloaded from this web page for public use. All NIST publications are available in the public domain according to Title 17 of the United States Code, however acknowledgement of the NVD when using our information is always appreciated.
Exploit-db indexed in a BleveDB

Used to lookup identified software products and search for applicable exploit PoC code.

Indexed in a bleve database.

Source: https://github.com/offensive-security/exploitdb

LICENSE: GPL-2

Further Licensing Details

The LICENSES file contains all licenses of data sources that provide one.
If you think that something should not be listed here please get in touch.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CloneDBs 

func CloneDBs(force bool)

CloneDBs will clone the data bases initially from the public git repository TODO: add windows support

func DownloadDBs added in v0.7.0 

func DownloadDBs(serverURL string, force bool) error

DownloadDBs downloads the latest databases from the configured server

func DownloadGeoLite 

func DownloadGeoLite()

DownloadGeoLite will download the GeoLite Database if the API key is set in the environment

func GenerateDBs 

func GenerateDBs(nvdIndexStartYear int)

GenerateDBs allows to fetch the databases from their initial sources and generate the preprocessed form that netcap uses

func IndexData 

func IndexData(in string, out string, buildPath string, nvdIndexStart int, verbose bool)

IndexData will index the data into a bleve database for full text search

func ListAvailableVersions added in v0.7.0 

func ListAvailableVersions(serverURL string) error

ListAvailableVersions lists all available database versions from the server

func UpdateDBs 

func UpdateDBs()

UpdateDBs will update the databases on disk by pulling from the public github repository

Types

type DBMetadata added in v0.7.0 

type DBMetadata struct {
	Version      string `json:"version"`
	CreatedAt    string `json:"created_at"`
	Tarball      string `json:"tarball"`
	NVDStartYear int    `json:"nvd_start_year"`
}

DBMetadata represents metadata about a database version

type DBServer added in v0.7.0 

type DBServer struct {
	// contains filtered or unexported fields
}

DBServer represents the database server

func NewDBServer added in v0.7.0 

func NewDBServer(addr string, nvdStartYear int, verbose bool) *DBServer

NewDBServer creates a new database server instance

func (*DBServer) Start added in v0.7.0 

func (s *DBServer) Start() error

Start starts the database server

type NVD2 added in v0.7.0 

type NVD2 struct {
	ResultsPerPage  int    `json:"resultsPerPage"`
	StartIndex      int    `json:"startIndex"`
	TotalResults    int    `json:"totalResults"`
	Format          string `json:"format"`
	Version         string `json:"version"`
	Timestamp       string `json:"timestamp"`
	Vulnerabilities []struct {
		Cve struct {
			ID               string        `json:"id"`
			SourceIdentifier string        `json:"sourceIdentifier"`
			Published        string        `json:"published"`
			LastModified     string        `json:"lastModified"`
			VulnStatus       string        `json:"vulnStatus"`
			CveTags          []interface{} `json:"cveTags"`
			Descriptions     []struct {
				Lang  string `json:"lang"`
				Value string `json:"value"`
			} `json:"descriptions"`
			Metrics struct {
				CvssMetricV2 []struct {
					Source   string `json:"source"`
					Type     string `json:"type"`
					CvssData struct {
						Version               string  `json:"version"`
						VectorString          string  `json:"vectorString"`
						BaseScore             float64 `json:"baseScore"`
						AccessVector          string  `json:"accessVector"`
						AccessComplexity      string  `json:"accessComplexity"`
						Authentication        string  `json:"authentication"`
						ConfidentialityImpact string  `json:"confidentialityImpact"`
						IntegrityImpact       string  `json:"integrityImpact"`
						AvailabilityImpact    string  `json:"availabilityImpact"`
					} `json:"cvssData"`
					BaseSeverity            string  `json:"baseSeverity"`
					ExploitabilityScore     float64 `json:"exploitabilityScore"`
					ImpactScore             float64 `json:"impactScore"`
					AcInsufInfo             bool    `json:"acInsufInfo"`
					ObtainAllPrivilege      bool    `json:"obtainAllPrivilege"`
					ObtainUserPrivilege     bool    `json:"obtainUserPrivilege"`
					ObtainOtherPrivilege    bool    `json:"obtainOtherPrivilege"`
					UserInteractionRequired bool    `json:"userInteractionRequired"`
				} `json:"cvssMetricV2"`
			} `json:"metrics"`
			Weaknesses []struct {
				Source      string `json:"source"`
				Type        string `json:"type"`
				Description []struct {
					Lang  string `json:"lang"`
					Value string `json:"value"`
				} `json:"description"`
			} `json:"weaknesses"`
			Configurations []struct {
				Nodes []struct {
					Operator string `json:"operator"`
					Negate   bool   `json:"negate"`
					CpeMatch []struct {
						Vulnerable            bool   `json:"vulnerable"`
						Criteria              string `json:"criteria"`
						MatchCriteriaID       string `json:"matchCriteriaId"`
						VersionStartIncluding string `json:"versionStartIncluding,omitempty"`
						VersionEndExcluding   string `json:"versionEndExcluding,omitempty"`
						VersionStartExcluding string `json:"versionStartExcluding,omitempty"`
						VersionEndIncluding   string `json:"versionEndIncluding,omitempty"`
					} `json:"cpeMatch"`
				} `json:"nodes"`
			} `json:"configurations"`
			References []struct {
				URL    string `json:"url"`
				Source string `json:"source"`
			} `json:"references"`
		} `json:"cve"`
	} `json:"vulnerabilities"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.