domainscan

package
v2.7.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 7, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Index

Constants

View Source
const (
	PostureSeverityHigh   = "high"
	PostureSeverityMedium = "medium"
	PostureSeverityLow    = "low"
)

Finding severities for posture checks

Variables

View Source
var DomainScanReportSchema = jsonx.SchemaFrom[Report]()

DomainScanReportSchema is the reflected JSON schema for DomainScanReport, the shape of a domain scan completion Notification's data

View Source
var (

	// ErrInvalidDomain is returned when its not able to determine the apex domain
	ErrInvalidDomain = errors.New("could not determine domain")
)

Functions

func BuildScanReport

func BuildScanReport(result *url_scanner.ScanGetResponse, enrichment Enrichment, nonVendorCategories, deniedVendorNames []string) map[string]any

BuildScanReport combines a Cloudflare URL Scanner result with the Enrichment gathered by GatherEnrichment into a single report unified vendors/technologies, assets, findings, meta, platform, systems, and compliance sections

func GatherStatusPage

func GatherStatusPage(ctx context.Context, domain string) string

GatherStatusPage finds a site's public status page.

A status page is a host rather than a path, and almost always status.<domain>, so that is probed first: one request settles it for most companies, with no dependence on the page linking it anywhere. Scanning the homepage's links is the fallback, for the company on a hosted provider such as statuspage.io or one using a different label

func IsCanonicalComplianceType

func IsCanonicalComplianceType(value string) bool

IsCanonicalComplianceType reports whether value names one of the recognized document types

func MatchesComplianceType

func MatchesComplianceType(extracted, canonical string) bool

MatchesComplianceType reports whether an extracted type denotes the given canonical one

func NormalizeComplianceType

func NormalizeComplianceType(value string) string

NormalizeComplianceType folds a document type onto its canonical form, tolerating case, spacing, hyphenation and the alternative names above

func RegistrableDomain

func RegistrableDomain(host string) (string, bool)

RegistrableDomain returns the registrable domain (eTLD+1) for host, e.g. "app.hubspot.com" -> "hubspot.com", so a subdomain can be matched against the vendor that owns its root domain

Types

type AgentReadinessAssessment

type AgentReadinessAssessment struct {
	// Level is the assessment's score
	Level int64 `json:"level"`
	// LevelName is the human-readable name for Level (e.g. "Bot-Aware")
	LevelName string `json:"level_name,omitempty"`
	// Reference links to Cloudflare's writeup of what the assessment measures
	Reference string `json:"reference,omitempty"`
	// TotalChecks is how many checks reported a status
	TotalChecks int `json:"total_checks"`
	// PassedChecks is how many of those checks passed
	PassedChecks int `json:"passed_checks"`
	// FailedChecks is how many of those checks failed
	FailedChecks int `json:"failed_checks"`
	// Checks are every check that reported a status, ordered by check path
	Checks []AgentReadinessCheck `json:"checks,omitempty"`
}

AgentReadinessAssessment is the full structured agent-readiness result for a domain, retained alongside the failure-only AgentReadinessFinding so a score can be derived from the pass ratio rather than inferred from the absence of failures

type AgentReadinessCheck

type AgentReadinessCheck struct {
	// Check is the dotted path identifying the check, e.g. "capabilities.mcp"
	Check string `json:"check"`
	// Status is the check's outcome as reported by Cloudflare, typically "pass" or "fail"
	Status string `json:"status"`
	// Message is the human-readable description of the outcome
	Message string `json:"message,omitempty"`
}

AgentReadinessCheck is one check within Cloudflare's agent-readiness assessment

type AgentReadinessFinding

type AgentReadinessFinding struct {
	// Level is the scan's agent-readiness score
	Level int64 `json:"level"`
	// LevelName is the human-readable name for Level (e.g. "Bot-Aware")
	LevelName string `json:"level_name"`
	// Checklist is a GitHub-flavored Markdown task list of the failing checks
	Checklist string `json:"checklist"`
	// Reference links to Cloudflare's writeup of what the assessment measures
	Reference string `json:"reference"`
	// Domain is the domain this finding was raised against, set only once merged across domains
	Domain string `json:"domain,omitempty"`
}

AgentReadinessFinding is one domain's agent-readiness assessment finding. Domain is only set once the finding has been merged across every domain in a submission

type Assets

type Assets struct {
	// DNSRecords are the DNS records resolved during the scan, plus any subdomains discovered by
	// the domainscan enrichment (type "internal")
	DNSRecords []DNSRecord `json:"dns_records,omitempty"`
	// IPAddresses are the IP addresses resolved during the scan
	IPAddresses []IPAddress `json:"ip_addresses,omitempty"`
}

Assets is the assets section of a scan report

func (Assets) IsEmpty

func (a Assets) IsEmpty() bool

IsEmpty reports whether none of the assets sections were populated

type BrandDesignProfile

type BrandDesignProfile struct {
	Error                    string `json:"error,omitempty"`
	LogoURL                  string `json:"logo_url,omitempty"`
	FaviconURL               string `json:"favicon_url,omitempty"`
	PrimaryColor             string `json:"primary_color,omitempty"`
	Font                     string `json:"font,omitempty"`
	ForegroundColor          string `json:"foreground_color,omitempty"`
	BackgroundColor          string `json:"background_color,omitempty"`
	AccentColor              string `json:"accent_color,omitempty"`
	SecondaryBackgroundColor string `json:"secondary_background_color,omitempty"`
	SecondaryForegroundColor string `json:"secondary_foreground_color,omitempty"`

	ApplyToPreviewTrustcenter bool `json:"-"`
	ApplyToLiveTrustcenter    bool `json:"-"`
}

BrandDesignProfile is the visual branding extracted from a rendered website

func (BrandDesignProfile) IsEmpty

func (b BrandDesignProfile) IsEmpty() bool

IsEmpty checks if branding data was found after the browser rendering job completes

type Branding

type Branding struct {
	Error string `json:"error,omitempty"`
	// Favicon is the scanned site's favicon
	Favicon Favicon `json:"favicon"`
	// LogoURL is the primary site logo discovered from the rendered page
	LogoURL string `json:"logo_url,omitempty"`
	// PrimaryColor is the site's primary brand color
	PrimaryColor string `json:"primary_color,omitempty"`
	// Font is the site's primary font family
	Font string `json:"font,omitempty"`
	// ForegroundColor is the main text or foreground color
	ForegroundColor string `json:"foreground_color,omitempty"`
	// BackgroundColor is the main page background color
	BackgroundColor string `json:"background_color,omitempty"`
	// AccentColor is the color used for calls to action, links, or highlights
	AccentColor string `json:"accent_color,omitempty"`
	// SecondaryBackgroundColor is the background color used for contrasting surfaces
	SecondaryBackgroundColor string `json:"secondary_background_color,omitempty"`
	// SecondaryForegroundColor is the text color used on secondary background surfaces
	SecondaryForegroundColor string `json:"secondary_foreground_color,omitempty"`
}

Branding is visual branding data captured from the scan, usable for org avatars, trust center logos, or similar presentational purposes

type CompanyProfile

type CompanyProfile struct {
	Name             string      `json:"name,omitempty"`
	Description      string      `json:"description,omitempty"`
	Industry         string      `json:"industry,omitempty"`
	Systems          []System    `json:"systems,omitempty"`
	Location         string      `json:"location,omitempty"`
	EmployeeRange    string      `json:"employee_range,omitempty"`
	FoundedYear      string      `json:"founded_year,omitempty"`
	EstimatedRevenue string      `json:"estimated_revenue,omitempty"`
	SocialLinks      SocialLinks `json:"social_links"`
	Customers        []string    `json:"customers,omitempty"`
	Technologies     []string    `json:"technologies,omitempty"`
	// ProvidedServices are the services or product categories the company itself provides
	// (e.g. "compliance automation", "payment processing", "CRM"), not third-party tools it uses
	ProvidedServices []string `json:"provided_services,omitempty"`
	// StatusPageURL is the company's public status/uptime page, if found
	StatusPageURL string `json:"status_page_url,omitempty"`
	// SubdomainLinks are other subdomains of the same company's domain linked
	// from the page (e.g. console.<domain>, app.<domain>, docs.<domain>)
	SubdomainLinks []string `json:"subdomain_links,omitempty"`
	// SSOSupported indicates the company's product advertises SSO support
	SSOSupported bool `json:"sso_supported,omitempty"`
	// MFASupported indicates the company's product advertises MFA support
	MFASupported bool `json:"mfa_supported,omitempty"`
	// SocialLoginSupported indicates the product advertises signing in via a third-party
	// social/consumer identity provider (e.g. "Sign in with Google", "Continue with GitHub")
	SocialLoginSupported bool `json:"social_login_supported,omitempty"`
	// CredentialsSupported indicates the product advertises traditional username/email and password authentication
	CredentialsSupported bool `json:"credentials_supported,omitempty"`
	// PasskeySupported indicates the product advertises passwordless authentication via passkeys, WebAuthn, or biometrics
	PasskeySupported bool `json:"passkey_supported,omitempty"`
}

CompanyProfile is the company information extracted from a website by the browser rendering AI

type Compliance

type Compliance struct {
	// Frameworks lists compliance frameworks or certifications mentioned (e.g., SOC 2, ISO 27001, GDPR, HIPAA)
	Frameworks []string `json:"frameworks,omitempty"`
	// IsSOC2 indicates the company claims SOC 2 (Type I or Type II) certification or compliance
	IsSOC2 bool `json:"is_soc2,omitempty"`
	// Controls lists individual security or compliance controls or practices mentioned
	Controls []string `json:"controls,omitempty"`
	// TrustCenterHostedBy is the platform serving the company's trust center, if one was found
	TrustCenterHostedBy string `json:"trust_center_hosted_by,omitempty"`
	// Documents lists compliance documents found on a trust center
	Documents []TrustDocument `json:"documents,omitempty"`
	// PageType categorizes the compliance page that was analyzed (e.g. privacy_policy, trust_center)
	PageType string `json:"page_type,omitempty"`
	// Links are the typed compliance document links found on the site. Unlike Documents,
	// whose Name is a free-text title, each link carries a machine-readable Type
	Links []ComplianceLink `json:"links,omitempty"`
}

Compliance is the company's compliance posture gathered by the domainscan enrichment

type ComplianceLink struct {
	// URL is the link target
	URL string `json:"url,omitempty"`
	// Type categorizes the linked document (e.g., privacy_policy, terms_of_service, trust_center, dpa, soc2_report, security, subprocessors, gdpr, cookie_policy, or other)
	Type string `json:"type,omitempty"`
}

ComplianceLink is a single compliance-related link found on a page, categorized by type

func GatherComplianceLinks(ctx context.Context, domain string) []ComplianceLink

GatherComplianceLinks finds a site's published legal documents by reading the links on its homepage, usually linked in the footer. domain may be a bare host or a URL

type CompliancePage

type CompliancePage struct {
	// URL is the page URL that was analyzed
	URL string `json:"url,omitempty"`
	// PageType categorizes the compliance document (e.g., privacy_policy, terms_of_service, trust_center, dpa, soc2_report, security, subprocessors, gdpr, cookie_policy)
	PageType string `json:"page_type,omitempty"`
	// Title is the page title
	Title string `json:"title,omitempty"`
	// Summary is a brief description of the page content
	Summary string `json:"summary,omitempty"`
	// Frameworks lists compliance frameworks or certifications mentioned (e.g., SOC 2, ISO 27001, GDPR, HIPAA)
	Frameworks []string `json:"frameworks,omitempty"`
	// SOC2Certified indicates whether the page claims SOC 2 (Type I or Type II) certification or compliance
	SOC2Certified bool `json:"soc2_certified,omitempty"`
	// LastUpdated is the last updated or effective date mentioned on the page
	LastUpdated string `json:"last_updated,omitempty"`
	// DownloadLinks contains URLs for downloadable reports or documents found on the page
	DownloadLinks []string `json:"download_links,omitempty"`
	// Subprocessors lists third-party vendors or sub-processors mentioned on the page
	Subprocessors []string `json:"subprocessors,omitempty"`
	// Controls lists individual security or compliance controls/practices mentioned, such as
	// those shown on a trust center page (e.g., MFA enforced, data encrypted at rest and in
	// transit, background checks performed, vulnerability scanning, penetration testing, incident response plan)
	Controls []string `json:"controls,omitempty"`
	// TrustCenterHostedBy is the platform serving the company's trust center, if
	// one was found (e.g., Vanta, Drata, SafeBase, Whistic, Conveyor, TrustArc,
	// or "self-hosted") — itself a vendor/technology signal for the company
	TrustCenterHostedBy string `json:"trust_center_hosted_by,omitempty"`
	// Documents lists compliance documents found on a trust center, each
	// flagged as public (directly accessible) or gated (requires a request/NDA)
	Documents []TrustDocument `json:"documents,omitempty"`
	// ComplianceLinks lists URLs to other compliance documents found on the page, each categorized by type
	ComplianceLinks []ComplianceLink `json:"compliance_links,omitempty"`
}

CompliancePage holds structured compliance information extracted from a single page

type Config

type Config struct {
	// APIToken use to authenticate into with the cloudflare API
	APIToken string
	// AccountID the APIToken is associated with
	AccountID string
	// CacheTTL is the cache TTL, in seconds, Cloudflare applies to Browser Rendering requests.
	// Zero disables caching for the request, forcing a fresh render instead of a cached one
	CacheTTL int
}

Config holds the Cloudflare credentials used for browser rendering and browser-derived enrichment lookups

func (*Config) GatherEnrichment

func (c *Config) GatherEnrichment(ctx context.Context, domain string, timeout time.Duration) (Enrichment, EnrichmentErrors)

GatherEnrichment runs the company profile, branding, compliance, and DNS vendor lookups for domain concurrently, bounded by timeout. Each lookup is best-effort

func (*Config) GetBrandingData

func (c *Config) GetBrandingData(ctx context.Context, domain string) (*BrandDesignProfile, error)

GetBrandingData extracts visual design tokens from the rendered website

func (*Config) GetCompanyData

func (c *Config) GetCompanyData(ctx context.Context, url string) (*CompanyProfile, error)

GetCompanyData builds a company profile from url's homepage, merged with whichever of companyProfileSubpaths resolve on the same host

func (*Config) GetComplianceData

func (c *Config) GetComplianceData(ctx context.Context, domain string) (*CompliancePage, error)

GetComplianceData fetches compliance information from the given domain and, if it can derive one, from a trust.<domain> subdomain as well

type DMARCPolicy

type DMARCPolicy string

DMARCPolicy is the enforcement policy from a DMARC record's p= or sp= tag

const (
	// DMARCPolicyNone requests reporting only, with no instruction to block
	DMARCPolicyNone DMARCPolicy = "none"
	// DMARCPolicyQuarantine asks receivers to treat failing mail as suspicious
	DMARCPolicyQuarantine DMARCPolicy = "quarantine"
	// DMARCPolicyReject asks receivers to refuse failing mail outright
	DMARCPolicyReject DMARCPolicy = "reject"
)

func ParseDMARCPolicy

func ParseDMARCPolicy(value string) (DMARCPolicy, bool)

ParseDMARCPolicy folds a p= or sp= tag value onto a DMARCPolicy, returning ok false when the value isn't a recognized policy so a malformed record doesn't score as enforcing

func (DMARCPolicy) Valid

func (p DMARCPolicy) Valid() bool

Valid reports whether the policy is one a receiver would honor

type DNSRecord

type DNSRecord struct {
	// Domain is the DNS record's hostname
	Domain string `json:"domain"`
	// Type is the DNS record type (e.g. "A", "NS")
	Type string `json:"type"`
	// Vendor is the vendor this hostname was attributed to, if known
	Vendor string `json:"vendor,omitempty"`
}

DNSRecord is one DNS record entry in a scan report's assets section

type DNSVendor

type DNSVendor struct {
	Name string `json:"name,omitempty"`
	URL  string `json:"url,omitempty"`
}

DNSVendor is a single vendor identified from a DNS record. URL is only populated when the signal pointed at an actual hostname (MX, SPF include, or CNAME target); DKIM selector and TXT verification-tag signals carry a name only, since the selector/tag itself isn't a domain

type DNSVendorInfo

type DNSVendorInfo struct {
	// SPFRecord is the raw SPF TXT record found at the domain's apex, if any
	SPFRecord string `json:"spf_record,omitempty"`
	// DMARCRecord is the raw DMARC TXT record found at _dmarc.<domain>, if any
	DMARCRecord string `json:"dmarc_record,omitempty"`
	// DMARCPolicy is the enforcement policy from the DMARC record's p= tag (none, quarantine, or reject)
	DMARCPolicy string `json:"dmarc_policy,omitempty"`
	// MXHosts lists the mail exchanger hostnames for the domain's apex, in preference order
	MXHosts []string `json:"mx_hosts,omitempty"`
	// NSHosts lists the authoritative nameserver hostnames for the domain's apex, revealing
	// the DNS hosting vendor (e.g. Cloudflare, Route 53, NS1)
	NSHosts []string `json:"ns_hosts,omitempty"`
	// TXTRecords lists every TXT record found at the domain's apex
	TXTRecords []string `json:"txt_records,omitempty"`
	// DKIMSelectors lists DKIM selector labels (see commonDKIMSelectors) found on the domain's apex
	DKIMSelectors []string `json:"dkim_selectors,omitempty"`
	// Subdomains holds MX, TXT, CNAME, and DKIM selector records found on conventional
	// mail/vendor subdomains (see commonSubdomains), probed up to two levels deep
	Subdomains []SubdomainDNSInfo `json:"subdomains,omitempty"`
	// Vendors lists vendors identified from the MX, SPF, TXT, CNAME, DKIM selector, and
	// NS records found at the apex and on Subdomains
	Vendors []DNSVendor `json:"vendors,omitempty"`
}

DNSVendorInfo holds DNS-derived signals about a domain: mail routing/authentication at the apex, records on conventional subdomains (see commonSubdomains), and vendors those point to

func GetDNSVendorInfo

func GetDNSVendorInfo(ctx context.Context, rawURL string) (*DNSVendorInfo, error)

GetDNSVendorInfo resolves MX/SPF/DMARC at the apex, probes commonSubdomains for MX/TXT/CNAME/DKIM, and derives vendor names from whatever's found rather than a maintained lookup table

type EmailAuth

type EmailAuth struct {
	// SPFRecord is the raw SPF TXT record found at the apex, if any
	SPFRecord string `json:"spf_record,omitempty"`
	// SPFPolicy is the qualifier on the record's all mechanism, empty when the record has none
	SPFPolicy SPFPolicy `json:"spf_policy,omitempty"`
	// DMARCRecord is the raw DMARC TXT record found at _dmarc.<domain>, if any
	DMARCRecord string `json:"dmarc_record,omitempty"`
	// DMARCPolicy is the enforcement policy from the p= tag, empty when absent or unrecognized
	DMARCPolicy DMARCPolicy `json:"dmarc_policy,omitempty"`
	// DMARCSubdomainPolicy is the sp= tag, which overrides DMARCPolicy for subdomains when present
	DMARCSubdomainPolicy DMARCPolicy `json:"dmarc_subdomain_policy,omitempty"`
	// DMARCPercentage is the pct= tag: the share of failing mail the policy is applied to.
	// A p=reject at pct=10 enforces on a tenth of messages, so policy alone overstates protection
	DMARCPercentage int `json:"dmarc_percentage,omitempty"`
	// DMARCReportingURIs are the rua= aggregate report destinations, absent when nobody is reading the reports
	DMARCReportingURIs []string `json:"dmarc_reporting_uris,omitempty"`
	// DKIMSelectors are the DKIM selector labels found at conventional names. Absence is
	// suggestive rather than conclusive, since only common selectors are probed
	DKIMSelectors []string `json:"dkim_selectors,omitempty"`
	// MXHosts are the apex mail exchangers, in preference order
	MXHosts []string `json:"mx_hosts,omitempty"`
}

EmailAuth is a domain's email authentication posture: the records that decide whether someone else can send mail that appears to come from it. Derived from the DNS enrichment, which is otherwise consumed for vendor attribution and not retained in the report

func (EmailAuth) Enforces

func (e EmailAuth) Enforces() bool

Enforces reports whether the DMARC policy actually instructs receivers to act on failing mail across all of it, rather than reporting only or sampling a fraction

type Enrichment

type Enrichment struct {
	// Company is the base company profile
	Company *CompanyProfile `json:"company,omitempty"`
	// Branding is visual design data extracted from the rendered website
	Branding *BrandDesignProfile `json:"branding,omitempty"`
	// Compliance is compliance specific data such as soc2 attestation, subprocessors, etc
	Compliance *CompliancePage `json:"compliance,omitempty"`
	// DNS includes DNS probed data
	DNS *DNSVendorInfo `json:"dns,omitempty"`
	// WellKnown includes the well-known file and transport security probes
	WellKnown *WellKnown `json:"well_known,omitempty"`
}

Enrichment holds the best-effort pkg/domainscan results for a single domain

func (Enrichment) IsEmpty

func (e Enrichment) IsEmpty() bool

IsEmpty reports whether no enrichment lookup produced any data

type EnrichmentErrors

type EnrichmentErrors struct {
	Company    error
	Branding   error
	Compliance error
	DNS        error
	WellKnown  error
}

EnrichmentErrors holds the per-lookup errors from GatherEnrichment, each nil on success

type Favicon

type Favicon struct {
	// URL is the favicon's URL
	URL string `json:"url,omitempty"`
	// Hash is the favicon's content hash
	Hash string `json:"hash,omitempty"`
}

Favicon is the scanned site's favicon

type Findings

type Findings struct {
	// SecurityViolations are the scan's overall verdict categories
	SecurityViolations []string `json:"security_violations,omitempty"`
	// Risks are the scan's overall verdict tags
	Risks []string `json:"risks,omitempty"`
	// IsMalicious reports whether the scan's overall verdict flagged the site as malicious
	IsMalicious bool `json:"is_malicious,omitempty"`
	// MissingComplianceLinks is a GitHub-flavored Markdown task list of expected compliance
	// document types not found on the site
	MissingComplianceLinks string `json:"missing_compliance_links,omitempty"`
	// AgentReadiness is the scan's agent-readiness assessment findings
	AgentReadiness []AgentReadinessFinding `json:"agent_readiness,omitempty"`
	// Posture are the failing email authentication and well-known file checks
	Posture []PostureFinding `json:"posture,omitempty"`
}

Findings is the findings section of a scan report

type Geolocation

type Geolocation struct {
	// City is the resolved city name
	City string `json:"city,omitempty"`
	// Country is the resolved country code
	Country string `json:"country,omitempty"`
	// CountryName is the resolved country's full name
	CountryName string `json:"country_name,omitempty"`
	// Region is the resolved region or state
	Region string `json:"region,omitempty"`
	// Latitude is the resolved latitude
	Latitude float64 `json:"latitude,omitempty"`
	// Longitude is the resolved longitude
	Longitude float64 `json:"longitude,omitempty"`
}

Geolocation is the scanned site's resolved geographic location

type IPAddress

type IPAddress struct {
	// Address is the resolved IP address
	Address string `json:"address"`
	// ASN is the autonomous system number announcing the address, if known
	ASN string `json:"asn,omitempty"`
	// Org is the organization associated with the ASN, if known
	Org string `json:"org,omitempty"`
}

IPAddress is one resolved IP entry in a scan report's assets section

type JSONSchema

type JSONSchema struct {
	Type       string                        `json:"type"`
	Properties map[string]JSONSchemaProperty `json:"properties"`
	Required   []string                      `json:"required,omitempty"`
}

JSONSchema is the JSON schema for structured extraction

type JSONSchemaProperty

type JSONSchemaProperty struct {
	Type        string                        `json:"type"`
	Description string                        `json:"description,omitempty"`
	Items       *JSONSchemaProperty           `json:"items,omitempty"`
	Properties  map[string]JSONSchemaProperty `json:"properties,omitempty"`
	MaxItems    int                           `json:"maxItems,omitempty"`
	MaxLength   int                           `json:"maxLength,omitempty"`
	UniqueItems bool                          `json:"uniqueItems,omitempty"`
	Enum        []string                      `json:"enum,omitempty"`
}

JSONSchemaProperty describes a single property in a JSON schema

type LLMsTxt

type LLMsTxt struct {
	// Present reports whether /llms.txt was served
	Present bool `json:"present"`
	// URL is where it was found
	URL string `json:"url,omitempty"`
	// Title is the H1 project or site name, the spec's only required section
	Title string `json:"title,omitempty"`
	// Summary is the blockquote summary immediately following the H1, if present
	Summary string `json:"summary,omitempty"`
	// SectionCount is the number of H2-delimited sections
	SectionCount int `json:"section_count,omitempty"`
	// FullPresent reports whether the companion /llms-full.txt was also served
	FullPresent bool `json:"full_present,omitempty"`
}

LLMsTxt is the parsed result of probing a domain for an llms.txt summary file

type Meta

type Meta struct {
	// Rank is the site's Cloudflare Radar rank, if known
	Rank int `json:"rank,omitempty"`
	// URLCategories are the scanned URL's detected categories
	URLCategories []string `json:"url_categories,omitempty"`
	// DomainCategories are the scanned domain's detected categories
	DomainCategories []string `json:"domain_categories,omitempty"`
	// Geolocation is the scanned site's resolved geographic location
	Geolocation *Geolocation `json:"geolocation,omitempty"`
}

Meta is scan-level metadata: radar rank, categories, and geolocation

func (Meta) IsEmpty

func (m Meta) IsEmpty() bool

IsEmpty reports whether none of the meta sections were populated

type Platform

type Platform struct {
	// Name is the company's name
	Name string `json:"name"`
	// Description is the company's description
	Description string `json:"description"`
	// Industry is the company's industry
	Industry string `json:"industry"`
	// Location is the company's headquarters location
	Location string `json:"location"`
	// EmployeeRange is the company's estimated employee count range
	EmployeeRange string `json:"employee_range"`
	// FoundedYear is the year the company was founded
	FoundedYear string `json:"founded_year"`
	// EstimatedRevenue is the company's estimated revenue
	EstimatedRevenue string `json:"estimated_revenue"`
	// SSOSupported indicates the company's product advertises SSO support
	SSOSupported bool `json:"sso_supported"`
	// MFASupported indicates the company's product advertises MFA support
	MFASupported bool `json:"mfa_supported"`
	// SocialLinks holds the company's social media and community profile URLs
	SocialLinks SocialLinks `json:"social_links"`
	// StatusPageURL is the company's public status/uptime page, if found
	StatusPageURL string `json:"status_page_url,omitempty"`
	// Customers are customer names or logos found on the site
	Customers []string `json:"customers,omitempty"`
	// ProvidedServices are the services or product categories the company itself provides
	ProvidedServices []string `json:"provided_services,omitempty"`
	// AuthMethods are the authentication methods the company's product advertises support for
	AuthMethods []string `json:"auth_methods,omitempty"`
}

Platform is the scanned company's profile, using field names that mirror Openlane's Platform object

type PostureFinding

type PostureFinding struct {
	// Check identifies the check, e.g. "dmarc" or "security_txt"
	Check string `json:"check"`
	// Title is the short headline of what failed
	Title string `json:"title"`
	// Description explains the failure and what to do about it
	Description string `json:"description"`
	// Severity is high, medium or low
	Severity string `json:"severity"`
	// Domain is the domain this finding was raised against, set only once merged across domains
	Domain string `json:"domain,omitempty"`
}

PostureFinding is one failing email authentication or well-known file check, shaped to import directly as a Finding record

type PromptType

type PromptType string

type Report

type Report struct {
	// Scans is one entry per domain in the originating submission, carrying its own identity and outcome
	Scans []Result `json:"scans"`
	// Vendors is the union of vendors detected across every completed domain, deduped by name
	Vendors []Vendor `json:"vendors,omitempty"`
	// Technologies is the union of technologies detected across every completed domain, deduped by name
	Technologies []Technology `json:"technologies,omitempty"`
	// Assets is the union of DNS records, IP addresses, and internal domains across every completed domain
	Assets *Assets `json:"assets,omitempty"`
	// Branding is the first non-empty branding section found across completed domains
	Branding *Branding `json:"branding,omitempty"`
	// Findings is the union of findings across every completed domain
	Findings Findings `json:"findings,omitempty"`
	// Meta is the first non-empty meta section found across completed domains
	Meta *Meta `json:"meta,omitempty"`
	// Platform is the first non-empty platform section found across completed domains
	Platform *Platform `json:"platform,omitempty"`
	// Systems is the union of systems detected across every completed domain, deduped by name
	Systems []SystemEntry `json:"systems,omitempty"`
	// Compliance is the first non-empty compliance section found across completed domains
	Compliance *Compliance `json:"compliance,omitempty"`
	// WellKnown is the first non-empty well-known section found across completed domains,
	// which for a parent-plus-subdomain submission is the parent's
	WellKnown *WellKnown `json:"well_known,omitempty"`
	// AgentReadiness is the first non-empty agent-readiness assessment found across completed domains
	AgentReadiness *AgentReadinessAssessment `json:"agent_readiness,omitempty"`
	// EmailAuth is the first non-empty email authentication section found across completed domains
	EmailAuth *EmailAuth `json:"email_auth,omitempty"`
}

Report is the JSON-schema-described payload attached to a domain scan completion Notification. Scans holds one entry per domain in the originating submission (a single entry for a one-off scan, one entry per domain for an org-settings batch) every other section is the union of that data across every successfully completed domain

func MergeReports

func MergeReports(results []Result, reports []ScanReportInput) Report

MergeReports combines one BuildScanReport output per domain (already vendor-enriched, as stored on each domain's own Scan.Metadata) into a single report

type ReportConfig

type ReportConfig struct {
	// BulkScan submits all domains through Cloudflare's bulk URL Scanner endpoint. When disabled,
	// domains are submitted sequentially through the single Scanner endpoint instead.
	// Not all plans have bulk scanning available. free tier e.g does not
	BulkScan bool `json:"bulkscan" koanf:"bulkscan" default:"true"`
	// NonVendorCategories lists wappalyzer categories treated as technologies
	// instead of vendors when building an onboarding domain scan report
	NonVendorCategories []string `` /* 145-byte string literal not displayed */
	// DeniedVendorNames lists vendor names to always exclude from an onboarding domain scan report's vendor list
	DeniedVendorNames []string `` /* 245-byte string literal not displayed */
	// ScanTTL is the cache TTL, in seconds, for Browser Rendering requests issued during domain scan enrichment
	ScanTTL int `json:"scanttl" koanf:"scanttl" default:"86400"`
}

ReportConfig configures how BuildScanReport classifies vendors versus technologies and which vendor names it always excludes

type ResponseFormat

type ResponseFormat struct {
	Type   string     `json:"type"`
	Schema JSONSchema `json:"json_schema"`
}

ResponseFormat specifies JSON schema extraction

type Result

type Result struct {
	// Domain is the scanned domain
	Domain string `json:"domain"`
	// InternalScanID is the Scan record id this result belongs to
	InternalScanID string `json:"internal_scan_id"`
	// ExternalScanID is the Cloudflare URL Scanner task id, present when the scan completed
	ExternalScanID string `json:"external_scan_id,omitempty"`
	// URL is the scanned URL, present when the scan completed
	URL string `json:"url,omitempty"`
	// Status is "completed" or "failed"
	Status string `json:"status"`
}

Result is one domain's outcome within a DomainScanReport

type RobotsTxt

type RobotsTxt struct {
	// Present reports whether a robots.txt was served
	Present bool `json:"present"`
	// ContentSignals are the parsed Content-Signal directives, e.g. {"ai-train": "no"}
	ContentSignals map[string]string `json:"content_signals,omitempty"`
	// DisallowedAICrawlers are the known AI crawlers fully disallowed (Disallow: /)
	DisallowedAICrawlers []string `json:"disallowed_ai_crawlers,omitempty"`
	// NamedAICrawlers are the known AI crawlers named at all, whether allowed or disallowed
	NamedAICrawlers []string `json:"named_ai_crawlers,omitempty"`
	// Sitemaps are the declared sitemap URLs
	Sitemaps []string `json:"sitemaps,omitempty"`
}

RobotsTxt is the parsed result of probing a domain for robots.txt, focused on what it says about AI crawlers rather than general crawl directives

func (RobotsTxt) StatesAIPosition

func (r RobotsTxt) StatesAIPosition() bool

StatesAIPosition reports whether the file expresses any position on AI crawlers, through either named crawler rules or Content-Signal directives

type SPFPolicy

type SPFPolicy string

SPFPolicy is the qualifier on an SPF record's "all" mechanism, which is what actually decides how a receiver treats mail from a host the record doesn't authorize

const (
	// SPFPolicyHardFail is "-all": unauthorized senders are explicitly not permitted
	SPFPolicyHardFail SPFPolicy = "hard_fail"
	// SPFPolicySoftFail is "~all": unauthorized senders should be accepted but marked
	SPFPolicySoftFail SPFPolicy = "soft_fail"
	// SPFPolicyNeutral is "?all": the record expresses no opinion, which is equivalent to no SPF
	SPFPolicyNeutral SPFPolicy = "neutral"
	// SPFPolicyPassAll is "+all": every host on the internet is authorized, effectively always a misconfiguration
	SPFPolicyPassAll SPFPolicy = "pass_all"
)

func (SPFPolicy) Valid

func (p SPFPolicy) Valid() bool

Valid reports whether the policy is one of the recognized qualifiers

type ScanReport

type ScanReport struct {
	// ExternalScanID is the Cloudflare URL Scanner task id
	ExternalScanID string `json:"external_scan_id,omitempty"`
	// URL is the scanned URL
	URL string `json:"url,omitempty"`
	// Vendors are the vendors detected during the scan
	Vendors []Vendor `json:"vendors,omitempty"`
	// Technologies are the technologies detected during the scan
	Technologies []Technology `json:"technologies,omitempty"`
	// Assets are the DNS records, IP addresses, and internal domains resolved during the scan
	Assets *Assets `json:"assets,omitempty"`
	// Branding is visual branding data captured from the scan
	Branding *Branding `json:"branding,omitempty"`
	// Findings are the scan's security verdict, agent-readiness assessment, and compliance gaps
	Findings Findings `json:"findings"`
	// Meta is scan-level metadata (radar rank, categories, geolocation)
	Meta *Meta `json:"meta,omitempty"`
	// Platform is the scanned company's profile
	Platform *Platform `json:"platform,omitempty"`
	// Systems are the company's systems discovered by the domainscan enrichment
	Systems []SystemEntry `json:"systems,omitempty"`
	// Compliance is the company's compliance posture gathered by the domainscan enrichment
	Compliance *Compliance `json:"compliance,omitempty"`
	// WellKnown is the well-known file and transport security posture probed directly at the domain
	WellKnown *WellKnown `json:"well_known,omitempty"`
	// AgentReadiness is the full structured agent-readiness assessment, present even when every check passed
	AgentReadiness *AgentReadinessAssessment `json:"agent_readiness,omitempty"`
	// EmailAuth is the domain's SPF, DKIM and DMARC posture
	EmailAuth *EmailAuth `json:"email_auth,omitempty"`
}

ScanReport is BuildScanReport's typed output for a single domain

type ScanReportInput

type ScanReportInput struct {
	// Domain is the domain that was scanned
	Domain string
	// Report is that domain's BuildScanReport output
	Report ScanReport
}

ScanReportInput pairs one domain with its typed scan report, for MergeReports. Domain is carried alongside the report (rather than embedded in it) since it identifies which scan produced the report, not something the report describes about itself

type SecurityTxt

type SecurityTxt struct {
	// Present reports whether a security.txt was served
	Present bool `json:"present"`
	// URL is where it was found, since the well-known path takes precedence over the legacy root path
	URL string `json:"url,omitempty"`
	// Contacts are the Contact field values, in the order published
	Contacts []string `json:"contacts,omitempty"`
	// Policy is the Policy field value, linking the vulnerability disclosure policy
	Policy string `json:"policy,omitempty"`
	// Encryption is the Encryption field value, linking a public key
	Encryption string `json:"encryption,omitempty"`
	// PreferredLanguages are the Preferred-Languages field values
	PreferredLanguages []string `json:"preferred_languages,omitempty"`
	// Expires is the Expires field value, after which RFC 9116 says the file should not be relied on
	Expires *time.Time `json:"expires,omitempty"`
	// Expired reports whether Expires is in the past
	Expired bool `json:"expired,omitempty"`
}

SecurityTxt is the parsed result of probing a domain for RFC 9116 security.txt

type SocialLinks struct {
	LinkedIn  string `json:"linkedin,omitempty"`
	Twitter   string `json:"twitter,omitempty"`
	GitHub    string `json:"github,omitempty"`
	Discord   string `json:"discord,omitempty"`
	Instagram string `json:"instagram,omitempty"`
	YouTube   string `json:"youtube,omitempty"`
	Facebook  string `json:"facebook,omitempty"`
}

SocialLinks holds a company's social media and community profile URLs

type SubdomainDNSInfo

type SubdomainDNSInfo struct {
	// Host is the fully probed hostname, e.g. "mail.example.com"
	Host string `json:"host,omitempty"`
	// MXHosts lists the mail exchanger hostnames found for this subdomain, if any
	MXHosts []string `json:"mx_hosts,omitempty"`
	// TXTRecords lists the TXT records found for this subdomain, if any
	TXTRecords []string `json:"txt_records,omitempty"`
	// CNAME is the canonical name this subdomain resolves to, if it's an
	// alias to another domain (e.g. a hosted checkout, status page, or
	// support portal pointed at a vendor's own domain)
	CNAME string `json:"cname,omitempty"`
	// DKIMSelectors lists DKIM selector labels (see commonDKIMSelectors)
	// found configured on this subdomain, e.g. "resend" for
	// "resend._domainkey.mail.example.com"
	DKIMSelectors []string `json:"dkim_selectors,omitempty"`
}

SubdomainDNSInfo holds the DNS records found for a single conventional subdomain probed alongside the apex

type System

type System struct {
	Name string `json:"name,omitempty"`
	// Summary is a brief, 1-2 sentence description of the system
	Summary string `json:"summary,omitempty"`
	// FullDescription is a more thorough description of the system, drawn
	// from documentation, architecture pages, or other technical content when available
	FullDescription string `json:"full_description,omitempty"`
}

System describes a single technical system or component that makes up the company's offering (e.g. a web console, public API, mobile app, or storage backend)

type SystemEntry

type SystemEntry struct {
	// SystemName is the system's name
	SystemName string `json:"system_name"`
	// Description is the system's description
	Description string `json:"description"`
}

SystemEntry is one of the company's systems, using field names that mirror Openlane's SystemDetail object

type Technology

type Technology struct {
	// Name is the technology's display name
	Name string `json:"name"`
	// URL is the technology's site or product URL, if known
	URL string `json:"url,omitempty"`
	// Categories are the technology's detected categories
	Categories []string `json:"categories,omitempty"`
}

Technology is one detected technology entry in a scan report

type TransportSecurity

type TransportSecurity struct {
	// HTTPSReachable reports whether the domain answered over HTTPS
	HTTPSReachable bool `json:"https_reachable"`
	// RedirectsToHTTPS reports whether a plain HTTP request ended up on HTTPS
	RedirectsToHTTPS bool `json:"redirects_to_https"`
	// HSTS reports whether a Strict-Transport-Security header was sent over HTTPS
	HSTS bool `json:"hsts"`
	// HSTSMaxAge is the header's max-age directive in seconds
	HSTSMaxAge int64 `json:"hsts_max_age,omitempty"`
	// HSTSIncludeSubdomains reports whether the header carried includeSubDomains
	HSTSIncludeSubdomains bool `json:"hsts_include_subdomains,omitempty"`
	// HSTSPreload reports whether the header carried the preload directive
	HSTSPreload bool `json:"hsts_preload,omitempty"`
}

TransportSecurity is the result of probing how strictly a domain enforces HTTPS

type TrustCenterPage

type TrustCenterPage struct {
	// HostedBy is the platform or vendor hosting this trust center (e.g.,
	// Vanta, Drata, SafeBase, Whistic, Conveyor, TrustArc, OneTrust), or
	// "self-hosted" if it appears to be a custom-built page
	HostedBy string `json:"hosted_by,omitempty"`
	// Frameworks lists compliance frameworks or certifications listed
	Frameworks []string `json:"frameworks,omitempty"`
	// SOC2Certified indicates whether the trust center claims SOC 2 (Type I or Type II) certification or compliance
	SOC2Certified bool `json:"soc2_certified,omitempty"`
	// Controls lists individual security or compliance controls or practices listed
	Controls []string `json:"controls,omitempty"`
	// Documents lists compliance documents or reports listed, each flagged public or gated
	Documents []TrustDocument `json:"documents,omitempty"`
	// Subprocessors lists third-party vendors or sub-processors mentioned
	Subprocessors []string `json:"subprocessors,omitempty"`
}

TrustCenterPage holds structured information extracted from a trust center or trust portal page

type TrustDocument

type TrustDocument struct {
	// Name is the document title (e.g., "SOC 2 Type II Report")
	Name string `json:"name,omitempty"`
	// URL is a direct link to the document, if one is available without requesting access
	URL string `json:"url,omitempty"`
	// Public indicates the document can be viewed or downloaded without requesting access or signing an NDA
	Public bool `json:"public,omitempty"`
}

TrustDocument is a single compliance document or report listed on a trust center

type Vendor

type Vendor struct {
	// Name is the vendor's canonical display name
	Name string `json:"name"`
	// LegalName is the vendor's raw, un-canonicalized legal entity name (e.g. "Cloudflare, Inc"),
	// set only when it differs from Name
	LegalName string `json:"legal_name,omitempty"`
	// URL is the vendor's site or product URL, if known
	URL string `json:"url,omitempty"`
	// Categories are the vendor's detected categories
	Categories []string `json:"categories,omitempty"`
}

Vendor is one detected vendor entry in a scan report

type WellKnown

type WellKnown struct {
	// SecurityTxt is the RFC 9116 security contact probe
	SecurityTxt *SecurityTxt `json:"security_txt,omitempty"`
	// Robots is the robots.txt AI crawler probe
	Robots *RobotsTxt `json:"robots_txt,omitempty"`
	// LLMs is the llms.txt probe
	LLMs *LLMsTxt `json:"llms_txt,omitempty"`
	// Transport is the HTTPS and HSTS enforcement probe
	Transport *TransportSecurity `json:"transport,omitempty"`
}

WellKnown holds the well-known file and transport security probes for a domain. Every field is best-effort: a nil section means that probe found nothing or failed, which for scoring purposes is treated the same as absent

func GetWellKnown

func GetWellKnown(ctx context.Context, domain string) (*WellKnown, error)

GetWellKnown probes domain for security.txt, robots.txt, llms.txt, and HTTPS enforcement, concurrently. Each probe is best-effort and independent, so a failure in one leaves the others intact; the returned error is non-nil only when domain itself could not be resolved to a registrable name

func (WellKnown) IsEmpty

func (w WellKnown) IsEmpty() bool

IsEmpty reports whether no probe produced a result

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL