Documentation
¶
Index ¶
- Constants
- Variables
- func BuildScanReport(result *url_scanner.ScanGetResponse, enrichment Enrichment, ...) map[string]any
- func GatherStatusPage(ctx context.Context, domain string) string
- func IsCanonicalComplianceType(value string) bool
- func MatchesComplianceType(extracted, canonical string) bool
- func NormalizeComplianceType(value string) string
- func RegistrableDomain(host string) (string, bool)
- type AgentReadinessAssessment
- type AgentReadinessCheck
- type AgentReadinessFinding
- type Assets
- type BrandDesignProfile
- type Branding
- type CompanyProfile
- type Compliance
- type ComplianceLink
- type CompliancePage
- type Config
- func (c *Config) GatherEnrichment(ctx context.Context, domain string, timeout time.Duration) (Enrichment, EnrichmentErrors)
- func (c *Config) GetBrandingData(ctx context.Context, domain string) (*BrandDesignProfile, error)
- func (c *Config) GetCompanyData(ctx context.Context, url string) (*CompanyProfile, error)
- func (c *Config) GetComplianceData(ctx context.Context, domain string) (*CompliancePage, error)
- type DMARCPolicy
- type DNSRecord
- type DNSVendor
- type DNSVendorInfo
- type EmailAuth
- type Enrichment
- type EnrichmentErrors
- type Favicon
- type Findings
- type Geolocation
- type IPAddress
- type JSONSchema
- type JSONSchemaProperty
- type LLMsTxt
- type Meta
- type Platform
- type PostureFinding
- type PromptType
- type Report
- type ReportConfig
- type ResponseFormat
- type Result
- type RobotsTxt
- type SPFPolicy
- type ScanReport
- type ScanReportInput
- type SecurityTxt
- type SocialLinks
- type SubdomainDNSInfo
- type System
- type SystemEntry
- type Technology
- type TransportSecurity
- type TrustCenterPage
- type TrustDocument
- type Vendor
- type WellKnown
Constants ¶
const ( PostureSeverityHigh = "high" PostureSeverityMedium = "medium" PostureSeverityLow = "low" )
Finding severities for posture checks
Variables ¶
var DomainScanReportSchema = jsonx.SchemaFrom[Report]()
DomainScanReportSchema is the reflected JSON schema for DomainScanReport, the shape of a domain scan completion Notification's data
var ( // ErrInvalidDomain is returned when its not able to determine the apex domain ErrInvalidDomain = errors.New("could not determine domain") )
Functions ¶
func BuildScanReport ¶
func BuildScanReport(result *url_scanner.ScanGetResponse, enrichment Enrichment, nonVendorCategories, deniedVendorNames []string) map[string]any
BuildScanReport combines a Cloudflare URL Scanner result with the Enrichment gathered by GatherEnrichment into a single report unified vendors/technologies, assets, findings, meta, platform, systems, and compliance sections
func GatherStatusPage ¶
GatherStatusPage finds a site's public status page.
A status page is a host rather than a path, and almost always status.<domain>, so that is probed first: one request settles it for most companies, with no dependence on the page linking it anywhere. Scanning the homepage's links is the fallback, for the company on a hosted provider such as statuspage.io or one using a different label
func IsCanonicalComplianceType ¶
IsCanonicalComplianceType reports whether value names one of the recognized document types
func MatchesComplianceType ¶
MatchesComplianceType reports whether an extracted type denotes the given canonical one
func NormalizeComplianceType ¶
NormalizeComplianceType folds a document type onto its canonical form, tolerating case, spacing, hyphenation and the alternative names above
func RegistrableDomain ¶
RegistrableDomain returns the registrable domain (eTLD+1) for host, e.g. "app.hubspot.com" -> "hubspot.com", so a subdomain can be matched against the vendor that owns its root domain
Types ¶
type AgentReadinessAssessment ¶
type AgentReadinessAssessment struct {
// Level is the assessment's score
Level int64 `json:"level"`
// LevelName is the human-readable name for Level (e.g. "Bot-Aware")
LevelName string `json:"level_name,omitempty"`
// Reference links to Cloudflare's writeup of what the assessment measures
Reference string `json:"reference,omitempty"`
// TotalChecks is how many checks reported a status
TotalChecks int `json:"total_checks"`
// PassedChecks is how many of those checks passed
PassedChecks int `json:"passed_checks"`
// FailedChecks is how many of those checks failed
FailedChecks int `json:"failed_checks"`
// Checks are every check that reported a status, ordered by check path
Checks []AgentReadinessCheck `json:"checks,omitempty"`
}
AgentReadinessAssessment is the full structured agent-readiness result for a domain, retained alongside the failure-only AgentReadinessFinding so a score can be derived from the pass ratio rather than inferred from the absence of failures
type AgentReadinessCheck ¶
type AgentReadinessCheck struct {
// Check is the dotted path identifying the check, e.g. "capabilities.mcp"
Check string `json:"check"`
// Status is the check's outcome as reported by Cloudflare, typically "pass" or "fail"
Status string `json:"status"`
// Message is the human-readable description of the outcome
Message string `json:"message,omitempty"`
}
AgentReadinessCheck is one check within Cloudflare's agent-readiness assessment
type AgentReadinessFinding ¶
type AgentReadinessFinding struct {
// Level is the scan's agent-readiness score
Level int64 `json:"level"`
// LevelName is the human-readable name for Level (e.g. "Bot-Aware")
LevelName string `json:"level_name"`
// Checklist is a GitHub-flavored Markdown task list of the failing checks
Checklist string `json:"checklist"`
// Reference links to Cloudflare's writeup of what the assessment measures
Reference string `json:"reference"`
// Domain is the domain this finding was raised against, set only once merged across domains
Domain string `json:"domain,omitempty"`
}
AgentReadinessFinding is one domain's agent-readiness assessment finding. Domain is only set once the finding has been merged across every domain in a submission
type Assets ¶
type Assets struct {
// DNSRecords are the DNS records resolved during the scan, plus any subdomains discovered by
// the domainscan enrichment (type "internal")
DNSRecords []DNSRecord `json:"dns_records,omitempty"`
// IPAddresses are the IP addresses resolved during the scan
IPAddresses []IPAddress `json:"ip_addresses,omitempty"`
}
Assets is the assets section of a scan report
type BrandDesignProfile ¶
type BrandDesignProfile struct {
Error string `json:"error,omitempty"`
LogoURL string `json:"logo_url,omitempty"`
FaviconURL string `json:"favicon_url,omitempty"`
PrimaryColor string `json:"primary_color,omitempty"`
Font string `json:"font,omitempty"`
ForegroundColor string `json:"foreground_color,omitempty"`
BackgroundColor string `json:"background_color,omitempty"`
AccentColor string `json:"accent_color,omitempty"`
SecondaryBackgroundColor string `json:"secondary_background_color,omitempty"`
SecondaryForegroundColor string `json:"secondary_foreground_color,omitempty"`
}
BrandDesignProfile is the visual branding extracted from a rendered website
func (BrandDesignProfile) IsEmpty ¶
func (b BrandDesignProfile) IsEmpty() bool
IsEmpty checks if branding data was found after the browser rendering job completes
type Branding ¶
type Branding struct {
Error string `json:"error,omitempty"`
// Favicon is the scanned site's favicon
Favicon Favicon `json:"favicon"`
// LogoURL is the primary site logo discovered from the rendered page
LogoURL string `json:"logo_url,omitempty"`
// PrimaryColor is the site's primary brand color
PrimaryColor string `json:"primary_color,omitempty"`
// Font is the site's primary font family
Font string `json:"font,omitempty"`
// ForegroundColor is the main text or foreground color
ForegroundColor string `json:"foreground_color,omitempty"`
// BackgroundColor is the main page background color
BackgroundColor string `json:"background_color,omitempty"`
// AccentColor is the color used for calls to action, links, or highlights
AccentColor string `json:"accent_color,omitempty"`
// SecondaryBackgroundColor is the background color used for contrasting surfaces
SecondaryBackgroundColor string `json:"secondary_background_color,omitempty"`
// SecondaryForegroundColor is the text color used on secondary background surfaces
SecondaryForegroundColor string `json:"secondary_foreground_color,omitempty"`
}
Branding is visual branding data captured from the scan, usable for org avatars, trust center logos, or similar presentational purposes
type CompanyProfile ¶
type CompanyProfile struct {
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Industry string `json:"industry,omitempty"`
Systems []System `json:"systems,omitempty"`
Location string `json:"location,omitempty"`
EmployeeRange string `json:"employee_range,omitempty"`
FoundedYear string `json:"founded_year,omitempty"`
EstimatedRevenue string `json:"estimated_revenue,omitempty"`
SocialLinks SocialLinks `json:"social_links"`
Customers []string `json:"customers,omitempty"`
Technologies []string `json:"technologies,omitempty"`
// ProvidedServices are the services or product categories the company itself provides
// (e.g. "compliance automation", "payment processing", "CRM"), not third-party tools it uses
ProvidedServices []string `json:"provided_services,omitempty"`
// StatusPageURL is the company's public status/uptime page, if found
StatusPageURL string `json:"status_page_url,omitempty"`
// SubdomainLinks are other subdomains of the same company's domain linked
// from the page (e.g. console.<domain>, app.<domain>, docs.<domain>)
SubdomainLinks []string `json:"subdomain_links,omitempty"`
// SSOSupported indicates the company's product advertises SSO support
SSOSupported bool `json:"sso_supported,omitempty"`
// MFASupported indicates the company's product advertises MFA support
MFASupported bool `json:"mfa_supported,omitempty"`
// SocialLoginSupported indicates the product advertises signing in via a third-party
// social/consumer identity provider (e.g. "Sign in with Google", "Continue with GitHub")
SocialLoginSupported bool `json:"social_login_supported,omitempty"`
// CredentialsSupported indicates the product advertises traditional username/email and password authentication
CredentialsSupported bool `json:"credentials_supported,omitempty"`
// PasskeySupported indicates the product advertises passwordless authentication via passkeys, WebAuthn, or biometrics
PasskeySupported bool `json:"passkey_supported,omitempty"`
}
CompanyProfile is the company information extracted from a website by the browser rendering AI
type Compliance ¶
type Compliance struct {
// Frameworks lists compliance frameworks or certifications mentioned (e.g., SOC 2, ISO 27001, GDPR, HIPAA)
Frameworks []string `json:"frameworks,omitempty"`
// IsSOC2 indicates the company claims SOC 2 (Type I or Type II) certification or compliance
IsSOC2 bool `json:"is_soc2,omitempty"`
// Controls lists individual security or compliance controls or practices mentioned
Controls []string `json:"controls,omitempty"`
// TrustCenterHostedBy is the platform serving the company's trust center, if one was found
TrustCenterHostedBy string `json:"trust_center_hosted_by,omitempty"`
// Documents lists compliance documents found on a trust center
Documents []TrustDocument `json:"documents,omitempty"`
// PageType categorizes the compliance page that was analyzed (e.g. privacy_policy, trust_center)
PageType string `json:"page_type,omitempty"`
// Links are the typed compliance document links found on the site. Unlike Documents,
// whose Name is a free-text title, each link carries a machine-readable Type
Links []ComplianceLink `json:"links,omitempty"`
}
Compliance is the company's compliance posture gathered by the domainscan enrichment
type ComplianceLink ¶
type ComplianceLink struct {
// URL is the link target
URL string `json:"url,omitempty"`
// Type categorizes the linked document (e.g., privacy_policy, terms_of_service, trust_center, dpa, soc2_report, security, subprocessors, gdpr, cookie_policy, or other)
Type string `json:"type,omitempty"`
}
ComplianceLink is a single compliance-related link found on a page, categorized by type
func GatherComplianceLinks ¶
func GatherComplianceLinks(ctx context.Context, domain string) []ComplianceLink
GatherComplianceLinks finds a site's published legal documents by reading the links on its homepage, usually linked in the footer. domain may be a bare host or a URL
type CompliancePage ¶
type CompliancePage struct {
// URL is the page URL that was analyzed
URL string `json:"url,omitempty"`
// PageType categorizes the compliance document (e.g., privacy_policy, terms_of_service, trust_center, dpa, soc2_report, security, subprocessors, gdpr, cookie_policy)
PageType string `json:"page_type,omitempty"`
// Title is the page title
Title string `json:"title,omitempty"`
// Summary is a brief description of the page content
Summary string `json:"summary,omitempty"`
// Frameworks lists compliance frameworks or certifications mentioned (e.g., SOC 2, ISO 27001, GDPR, HIPAA)
Frameworks []string `json:"frameworks,omitempty"`
// SOC2Certified indicates whether the page claims SOC 2 (Type I or Type II) certification or compliance
SOC2Certified bool `json:"soc2_certified,omitempty"`
// LastUpdated is the last updated or effective date mentioned on the page
LastUpdated string `json:"last_updated,omitempty"`
// DownloadLinks contains URLs for downloadable reports or documents found on the page
DownloadLinks []string `json:"download_links,omitempty"`
// Subprocessors lists third-party vendors or sub-processors mentioned on the page
Subprocessors []string `json:"subprocessors,omitempty"`
// Controls lists individual security or compliance controls/practices mentioned, such as
// those shown on a trust center page (e.g., MFA enforced, data encrypted at rest and in
// transit, background checks performed, vulnerability scanning, penetration testing, incident response plan)
Controls []string `json:"controls,omitempty"`
// TrustCenterHostedBy is the platform serving the company's trust center, if
// one was found (e.g., Vanta, Drata, SafeBase, Whistic, Conveyor, TrustArc,
// or "self-hosted") — itself a vendor/technology signal for the company
TrustCenterHostedBy string `json:"trust_center_hosted_by,omitempty"`
// Documents lists compliance documents found on a trust center, each
// flagged as public (directly accessible) or gated (requires a request/NDA)
Documents []TrustDocument `json:"documents,omitempty"`
// ComplianceLinks lists URLs to other compliance documents found on the page, each categorized by type
ComplianceLinks []ComplianceLink `json:"compliance_links,omitempty"`
}
CompliancePage holds structured compliance information extracted from a single page
type Config ¶
type Config struct {
// APIToken use to authenticate into with the cloudflare API
APIToken string
// AccountID the APIToken is associated with
AccountID string
// CacheTTL is the cache TTL, in seconds, Cloudflare applies to Browser Rendering requests.
// Zero disables caching for the request, forcing a fresh render instead of a cached one
CacheTTL int
}
Config holds the Cloudflare credentials used for browser rendering and browser-derived enrichment lookups
func (*Config) GatherEnrichment ¶
func (c *Config) GatherEnrichment(ctx context.Context, domain string, timeout time.Duration) (Enrichment, EnrichmentErrors)
GatherEnrichment runs the company profile, branding, compliance, and DNS vendor lookups for domain concurrently, bounded by timeout. Each lookup is best-effort
func (*Config) GetBrandingData ¶
GetBrandingData extracts visual design tokens from the rendered website
func (*Config) GetCompanyData ¶
GetCompanyData builds a company profile from url's homepage, merged with whichever of companyProfileSubpaths resolve on the same host
func (*Config) GetComplianceData ¶
GetComplianceData fetches compliance information from the given domain and, if it can derive one, from a trust.<domain> subdomain as well
type DMARCPolicy ¶
type DMARCPolicy string
DMARCPolicy is the enforcement policy from a DMARC record's p= or sp= tag
const ( // DMARCPolicyNone requests reporting only, with no instruction to block DMARCPolicyNone DMARCPolicy = "none" // DMARCPolicyQuarantine asks receivers to treat failing mail as suspicious DMARCPolicyQuarantine DMARCPolicy = "quarantine" // DMARCPolicyReject asks receivers to refuse failing mail outright DMARCPolicyReject DMARCPolicy = "reject" )
func ParseDMARCPolicy ¶
func ParseDMARCPolicy(value string) (DMARCPolicy, bool)
ParseDMARCPolicy folds a p= or sp= tag value onto a DMARCPolicy, returning ok false when the value isn't a recognized policy so a malformed record doesn't score as enforcing
func (DMARCPolicy) Valid ¶
func (p DMARCPolicy) Valid() bool
Valid reports whether the policy is one a receiver would honor
type DNSRecord ¶
type DNSRecord struct {
// Domain is the DNS record's hostname
Domain string `json:"domain"`
// Type is the DNS record type (e.g. "A", "NS")
Type string `json:"type"`
// Vendor is the vendor this hostname was attributed to, if known
Vendor string `json:"vendor,omitempty"`
}
DNSRecord is one DNS record entry in a scan report's assets section
type DNSVendor ¶
DNSVendor is a single vendor identified from a DNS record. URL is only populated when the signal pointed at an actual hostname (MX, SPF include, or CNAME target); DKIM selector and TXT verification-tag signals carry a name only, since the selector/tag itself isn't a domain
type DNSVendorInfo ¶
type DNSVendorInfo struct {
// SPFRecord is the raw SPF TXT record found at the domain's apex, if any
SPFRecord string `json:"spf_record,omitempty"`
// DMARCRecord is the raw DMARC TXT record found at _dmarc.<domain>, if any
DMARCRecord string `json:"dmarc_record,omitempty"`
// DMARCPolicy is the enforcement policy from the DMARC record's p= tag (none, quarantine, or reject)
DMARCPolicy string `json:"dmarc_policy,omitempty"`
// MXHosts lists the mail exchanger hostnames for the domain's apex, in preference order
MXHosts []string `json:"mx_hosts,omitempty"`
// NSHosts lists the authoritative nameserver hostnames for the domain's apex, revealing
// the DNS hosting vendor (e.g. Cloudflare, Route 53, NS1)
NSHosts []string `json:"ns_hosts,omitempty"`
// TXTRecords lists every TXT record found at the domain's apex
TXTRecords []string `json:"txt_records,omitempty"`
// DKIMSelectors lists DKIM selector labels (see commonDKIMSelectors) found on the domain's apex
DKIMSelectors []string `json:"dkim_selectors,omitempty"`
// Subdomains holds MX, TXT, CNAME, and DKIM selector records found on conventional
// mail/vendor subdomains (see commonSubdomains), probed up to two levels deep
Subdomains []SubdomainDNSInfo `json:"subdomains,omitempty"`
// Vendors lists vendors identified from the MX, SPF, TXT, CNAME, DKIM selector, and
// NS records found at the apex and on Subdomains
Vendors []DNSVendor `json:"vendors,omitempty"`
}
DNSVendorInfo holds DNS-derived signals about a domain: mail routing/authentication at the apex, records on conventional subdomains (see commonSubdomains), and vendors those point to
func GetDNSVendorInfo ¶
func GetDNSVendorInfo(ctx context.Context, rawURL string) (*DNSVendorInfo, error)
GetDNSVendorInfo resolves MX/SPF/DMARC at the apex, probes commonSubdomains for MX/TXT/CNAME/DKIM, and derives vendor names from whatever's found rather than a maintained lookup table
type EmailAuth ¶
type EmailAuth struct {
// SPFRecord is the raw SPF TXT record found at the apex, if any
SPFRecord string `json:"spf_record,omitempty"`
// SPFPolicy is the qualifier on the record's all mechanism, empty when the record has none
SPFPolicy SPFPolicy `json:"spf_policy,omitempty"`
// DMARCRecord is the raw DMARC TXT record found at _dmarc.<domain>, if any
DMARCRecord string `json:"dmarc_record,omitempty"`
// DMARCPolicy is the enforcement policy from the p= tag, empty when absent or unrecognized
DMARCPolicy DMARCPolicy `json:"dmarc_policy,omitempty"`
// DMARCSubdomainPolicy is the sp= tag, which overrides DMARCPolicy for subdomains when present
DMARCSubdomainPolicy DMARCPolicy `json:"dmarc_subdomain_policy,omitempty"`
// DMARCPercentage is the pct= tag: the share of failing mail the policy is applied to.
// A p=reject at pct=10 enforces on a tenth of messages, so policy alone overstates protection
DMARCPercentage int `json:"dmarc_percentage,omitempty"`
// DMARCReportingURIs are the rua= aggregate report destinations, absent when nobody is reading the reports
DMARCReportingURIs []string `json:"dmarc_reporting_uris,omitempty"`
// DKIMSelectors are the DKIM selector labels found at conventional names. Absence is
// suggestive rather than conclusive, since only common selectors are probed
DKIMSelectors []string `json:"dkim_selectors,omitempty"`
// MXHosts are the apex mail exchangers, in preference order
MXHosts []string `json:"mx_hosts,omitempty"`
}
EmailAuth is a domain's email authentication posture: the records that decide whether someone else can send mail that appears to come from it. Derived from the DNS enrichment, which is otherwise consumed for vendor attribution and not retained in the report
type Enrichment ¶
type Enrichment struct {
// Company is the base company profile
Company *CompanyProfile `json:"company,omitempty"`
// Branding is visual design data extracted from the rendered website
Branding *BrandDesignProfile `json:"branding,omitempty"`
// Compliance is compliance specific data such as soc2 attestation, subprocessors, etc
Compliance *CompliancePage `json:"compliance,omitempty"`
// DNS includes DNS probed data
DNS *DNSVendorInfo `json:"dns,omitempty"`
// WellKnown includes the well-known file and transport security probes
WellKnown *WellKnown `json:"well_known,omitempty"`
}
Enrichment holds the best-effort pkg/domainscan results for a single domain
func (Enrichment) IsEmpty ¶
func (e Enrichment) IsEmpty() bool
IsEmpty reports whether no enrichment lookup produced any data
type EnrichmentErrors ¶
type EnrichmentErrors struct {
Company error
Branding error
Compliance error
DNS error
WellKnown error
}
EnrichmentErrors holds the per-lookup errors from GatherEnrichment, each nil on success
type Favicon ¶
type Favicon struct {
// URL is the favicon's URL
URL string `json:"url,omitempty"`
// Hash is the favicon's content hash
Hash string `json:"hash,omitempty"`
}
Favicon is the scanned site's favicon
type Findings ¶
type Findings struct {
// SecurityViolations are the scan's overall verdict categories
SecurityViolations []string `json:"security_violations,omitempty"`
// Risks are the scan's overall verdict tags
Risks []string `json:"risks,omitempty"`
// IsMalicious reports whether the scan's overall verdict flagged the site as malicious
IsMalicious bool `json:"is_malicious,omitempty"`
// MissingComplianceLinks is a GitHub-flavored Markdown task list of expected compliance
// document types not found on the site
MissingComplianceLinks string `json:"missing_compliance_links,omitempty"`
// AgentReadiness is the scan's agent-readiness assessment findings
AgentReadiness []AgentReadinessFinding `json:"agent_readiness,omitempty"`
// Posture are the failing email authentication and well-known file checks
Posture []PostureFinding `json:"posture,omitempty"`
}
Findings is the findings section of a scan report
type Geolocation ¶
type Geolocation struct {
// City is the resolved city name
City string `json:"city,omitempty"`
// Country is the resolved country code
Country string `json:"country,omitempty"`
// CountryName is the resolved country's full name
CountryName string `json:"country_name,omitempty"`
// Region is the resolved region or state
Region string `json:"region,omitempty"`
// Latitude is the resolved latitude
Latitude float64 `json:"latitude,omitempty"`
// Longitude is the resolved longitude
Longitude float64 `json:"longitude,omitempty"`
}
Geolocation is the scanned site's resolved geographic location
type IPAddress ¶
type IPAddress struct {
// Address is the resolved IP address
Address string `json:"address"`
// ASN is the autonomous system number announcing the address, if known
ASN string `json:"asn,omitempty"`
// Org is the organization associated with the ASN, if known
Org string `json:"org,omitempty"`
}
IPAddress is one resolved IP entry in a scan report's assets section
type JSONSchema ¶
type JSONSchema struct {
Type string `json:"type"`
Properties map[string]JSONSchemaProperty `json:"properties"`
Required []string `json:"required,omitempty"`
}
JSONSchema is the JSON schema for structured extraction
type JSONSchemaProperty ¶
type JSONSchemaProperty struct {
Type string `json:"type"`
Description string `json:"description,omitempty"`
Items *JSONSchemaProperty `json:"items,omitempty"`
Properties map[string]JSONSchemaProperty `json:"properties,omitempty"`
MaxItems int `json:"maxItems,omitempty"`
MaxLength int `json:"maxLength,omitempty"`
UniqueItems bool `json:"uniqueItems,omitempty"`
Enum []string `json:"enum,omitempty"`
}
JSONSchemaProperty describes a single property in a JSON schema
type LLMsTxt ¶
type LLMsTxt struct {
// Present reports whether /llms.txt was served
Present bool `json:"present"`
// URL is where it was found
URL string `json:"url,omitempty"`
// Title is the H1 project or site name, the spec's only required section
Title string `json:"title,omitempty"`
// Summary is the blockquote summary immediately following the H1, if present
Summary string `json:"summary,omitempty"`
// SectionCount is the number of H2-delimited sections
SectionCount int `json:"section_count,omitempty"`
// FullPresent reports whether the companion /llms-full.txt was also served
FullPresent bool `json:"full_present,omitempty"`
}
LLMsTxt is the parsed result of probing a domain for an llms.txt summary file
type Meta ¶
type Meta struct {
// Rank is the site's Cloudflare Radar rank, if known
Rank int `json:"rank,omitempty"`
// URLCategories are the scanned URL's detected categories
URLCategories []string `json:"url_categories,omitempty"`
// DomainCategories are the scanned domain's detected categories
DomainCategories []string `json:"domain_categories,omitempty"`
// Geolocation is the scanned site's resolved geographic location
Geolocation *Geolocation `json:"geolocation,omitempty"`
}
Meta is scan-level metadata: radar rank, categories, and geolocation
type Platform ¶
type Platform struct {
// Name is the company's name
Name string `json:"name"`
// Description is the company's description
Description string `json:"description"`
// Industry is the company's industry
Industry string `json:"industry"`
// Location is the company's headquarters location
Location string `json:"location"`
// EmployeeRange is the company's estimated employee count range
EmployeeRange string `json:"employee_range"`
// FoundedYear is the year the company was founded
FoundedYear string `json:"founded_year"`
// EstimatedRevenue is the company's estimated revenue
EstimatedRevenue string `json:"estimated_revenue"`
// SSOSupported indicates the company's product advertises SSO support
SSOSupported bool `json:"sso_supported"`
// MFASupported indicates the company's product advertises MFA support
MFASupported bool `json:"mfa_supported"`
// SocialLinks holds the company's social media and community profile URLs
SocialLinks SocialLinks `json:"social_links"`
// StatusPageURL is the company's public status/uptime page, if found
StatusPageURL string `json:"status_page_url,omitempty"`
// Customers are customer names or logos found on the site
Customers []string `json:"customers,omitempty"`
// ProvidedServices are the services or product categories the company itself provides
ProvidedServices []string `json:"provided_services,omitempty"`
// AuthMethods are the authentication methods the company's product advertises support for
AuthMethods []string `json:"auth_methods,omitempty"`
}
Platform is the scanned company's profile, using field names that mirror Openlane's Platform object
type PostureFinding ¶
type PostureFinding struct {
// Check identifies the check, e.g. "dmarc" or "security_txt"
Check string `json:"check"`
// Title is the short headline of what failed
Title string `json:"title"`
// Description explains the failure and what to do about it
Description string `json:"description"`
// Severity is high, medium or low
Severity string `json:"severity"`
// Domain is the domain this finding was raised against, set only once merged across domains
Domain string `json:"domain,omitempty"`
}
PostureFinding is one failing email authentication or well-known file check, shaped to import directly as a Finding record
type PromptType ¶
type PromptType string
type Report ¶
type Report struct {
// Scans is one entry per domain in the originating submission, carrying its own identity and outcome
Scans []Result `json:"scans"`
// Vendors is the union of vendors detected across every completed domain, deduped by name
Vendors []Vendor `json:"vendors,omitempty"`
// Technologies is the union of technologies detected across every completed domain, deduped by name
Technologies []Technology `json:"technologies,omitempty"`
// Assets is the union of DNS records, IP addresses, and internal domains across every completed domain
Assets *Assets `json:"assets,omitempty"`
// Branding is the first non-empty branding section found across completed domains
Branding *Branding `json:"branding,omitempty"`
// Findings is the union of findings across every completed domain
Findings Findings `json:"findings,omitempty"`
// Meta is the first non-empty meta section found across completed domains
Meta *Meta `json:"meta,omitempty"`
// Platform is the first non-empty platform section found across completed domains
Platform *Platform `json:"platform,omitempty"`
// Systems is the union of systems detected across every completed domain, deduped by name
Systems []SystemEntry `json:"systems,omitempty"`
// Compliance is the first non-empty compliance section found across completed domains
Compliance *Compliance `json:"compliance,omitempty"`
// WellKnown is the first non-empty well-known section found across completed domains,
// which for a parent-plus-subdomain submission is the parent's
WellKnown *WellKnown `json:"well_known,omitempty"`
// AgentReadiness is the first non-empty agent-readiness assessment found across completed domains
AgentReadiness *AgentReadinessAssessment `json:"agent_readiness,omitempty"`
// EmailAuth is the first non-empty email authentication section found across completed domains
EmailAuth *EmailAuth `json:"email_auth,omitempty"`
}
Report is the JSON-schema-described payload attached to a domain scan completion Notification. Scans holds one entry per domain in the originating submission (a single entry for a one-off scan, one entry per domain for an org-settings batch) every other section is the union of that data across every successfully completed domain
func MergeReports ¶
func MergeReports(results []Result, reports []ScanReportInput) Report
MergeReports combines one BuildScanReport output per domain (already vendor-enriched, as stored on each domain's own Scan.Metadata) into a single report
type ReportConfig ¶
type ReportConfig struct {
// BulkScan submits all domains through Cloudflare's bulk URL Scanner endpoint. When disabled,
// domains are submitted sequentially through the single Scanner endpoint instead.
// Not all plans have bulk scanning available. free tier e.g does not
BulkScan bool `json:"bulkscan" koanf:"bulkscan" default:"true"`
// NonVendorCategories lists wappalyzer categories treated as technologies
// instead of vendors when building an onboarding domain scan report
NonVendorCategories []string `` /* 145-byte string literal not displayed */
// DeniedVendorNames lists vendor names to always exclude from an onboarding domain scan report's vendor list
DeniedVendorNames []string `` /* 245-byte string literal not displayed */
// ScanTTL is the cache TTL, in seconds, for Browser Rendering requests issued during domain scan enrichment
ScanTTL int `json:"scanttl" koanf:"scanttl" default:"86400"`
}
ReportConfig configures how BuildScanReport classifies vendors versus technologies and which vendor names it always excludes
type ResponseFormat ¶
type ResponseFormat struct {
Type string `json:"type"`
Schema JSONSchema `json:"json_schema"`
}
ResponseFormat specifies JSON schema extraction
type Result ¶
type Result struct {
// Domain is the scanned domain
Domain string `json:"domain"`
// InternalScanID is the Scan record id this result belongs to
InternalScanID string `json:"internal_scan_id"`
// ExternalScanID is the Cloudflare URL Scanner task id, present when the scan completed
ExternalScanID string `json:"external_scan_id,omitempty"`
// URL is the scanned URL, present when the scan completed
URL string `json:"url,omitempty"`
// Status is "completed" or "failed"
Status string `json:"status"`
}
Result is one domain's outcome within a DomainScanReport
type RobotsTxt ¶
type RobotsTxt struct {
// Present reports whether a robots.txt was served
Present bool `json:"present"`
// ContentSignals are the parsed Content-Signal directives, e.g. {"ai-train": "no"}
ContentSignals map[string]string `json:"content_signals,omitempty"`
// DisallowedAICrawlers are the known AI crawlers fully disallowed (Disallow: /)
DisallowedAICrawlers []string `json:"disallowed_ai_crawlers,omitempty"`
// NamedAICrawlers are the known AI crawlers named at all, whether allowed or disallowed
NamedAICrawlers []string `json:"named_ai_crawlers,omitempty"`
// Sitemaps are the declared sitemap URLs
Sitemaps []string `json:"sitemaps,omitempty"`
}
RobotsTxt is the parsed result of probing a domain for robots.txt, focused on what it says about AI crawlers rather than general crawl directives
func (RobotsTxt) StatesAIPosition ¶
StatesAIPosition reports whether the file expresses any position on AI crawlers, through either named crawler rules or Content-Signal directives
type SPFPolicy ¶
type SPFPolicy string
SPFPolicy is the qualifier on an SPF record's "all" mechanism, which is what actually decides how a receiver treats mail from a host the record doesn't authorize
const ( // SPFPolicyHardFail is "-all": unauthorized senders are explicitly not permitted SPFPolicyHardFail SPFPolicy = "hard_fail" // SPFPolicySoftFail is "~all": unauthorized senders should be accepted but marked SPFPolicySoftFail SPFPolicy = "soft_fail" // SPFPolicyNeutral is "?all": the record expresses no opinion, which is equivalent to no SPF SPFPolicyNeutral SPFPolicy = "neutral" // SPFPolicyPassAll is "+all": every host on the internet is authorized, effectively always a misconfiguration SPFPolicyPassAll SPFPolicy = "pass_all" )
type ScanReport ¶
type ScanReport struct {
// ExternalScanID is the Cloudflare URL Scanner task id
ExternalScanID string `json:"external_scan_id,omitempty"`
// URL is the scanned URL
URL string `json:"url,omitempty"`
// Vendors are the vendors detected during the scan
Vendors []Vendor `json:"vendors,omitempty"`
// Technologies are the technologies detected during the scan
Technologies []Technology `json:"technologies,omitempty"`
// Assets are the DNS records, IP addresses, and internal domains resolved during the scan
Assets *Assets `json:"assets,omitempty"`
// Branding is visual branding data captured from the scan
Branding *Branding `json:"branding,omitempty"`
// Findings are the scan's security verdict, agent-readiness assessment, and compliance gaps
Findings Findings `json:"findings"`
// Meta is scan-level metadata (radar rank, categories, geolocation)
Meta *Meta `json:"meta,omitempty"`
// Platform is the scanned company's profile
Platform *Platform `json:"platform,omitempty"`
// Systems are the company's systems discovered by the domainscan enrichment
Systems []SystemEntry `json:"systems,omitempty"`
// Compliance is the company's compliance posture gathered by the domainscan enrichment
Compliance *Compliance `json:"compliance,omitempty"`
// WellKnown is the well-known file and transport security posture probed directly at the domain
WellKnown *WellKnown `json:"well_known,omitempty"`
// AgentReadiness is the full structured agent-readiness assessment, present even when every check passed
AgentReadiness *AgentReadinessAssessment `json:"agent_readiness,omitempty"`
// EmailAuth is the domain's SPF, DKIM and DMARC posture
EmailAuth *EmailAuth `json:"email_auth,omitempty"`
}
ScanReport is BuildScanReport's typed output for a single domain
type ScanReportInput ¶
type ScanReportInput struct {
// Domain is the domain that was scanned
Domain string
// Report is that domain's BuildScanReport output
Report ScanReport
}
ScanReportInput pairs one domain with its typed scan report, for MergeReports. Domain is carried alongside the report (rather than embedded in it) since it identifies which scan produced the report, not something the report describes about itself
type SecurityTxt ¶
type SecurityTxt struct {
// Present reports whether a security.txt was served
Present bool `json:"present"`
// URL is where it was found, since the well-known path takes precedence over the legacy root path
URL string `json:"url,omitempty"`
// Contacts are the Contact field values, in the order published
Contacts []string `json:"contacts,omitempty"`
// Policy is the Policy field value, linking the vulnerability disclosure policy
Policy string `json:"policy,omitempty"`
// Encryption is the Encryption field value, linking a public key
Encryption string `json:"encryption,omitempty"`
// PreferredLanguages are the Preferred-Languages field values
PreferredLanguages []string `json:"preferred_languages,omitempty"`
// Expires is the Expires field value, after which RFC 9116 says the file should not be relied on
Expires *time.Time `json:"expires,omitempty"`
// Expired reports whether Expires is in the past
Expired bool `json:"expired,omitempty"`
}
SecurityTxt is the parsed result of probing a domain for RFC 9116 security.txt
type SocialLinks ¶
type SocialLinks struct {
LinkedIn string `json:"linkedin,omitempty"`
Twitter string `json:"twitter,omitempty"`
GitHub string `json:"github,omitempty"`
Discord string `json:"discord,omitempty"`
Instagram string `json:"instagram,omitempty"`
YouTube string `json:"youtube,omitempty"`
Facebook string `json:"facebook,omitempty"`
}
SocialLinks holds a company's social media and community profile URLs
type SubdomainDNSInfo ¶
type SubdomainDNSInfo struct {
// Host is the fully probed hostname, e.g. "mail.example.com"
Host string `json:"host,omitempty"`
// MXHosts lists the mail exchanger hostnames found for this subdomain, if any
MXHosts []string `json:"mx_hosts,omitempty"`
// TXTRecords lists the TXT records found for this subdomain, if any
TXTRecords []string `json:"txt_records,omitempty"`
// CNAME is the canonical name this subdomain resolves to, if it's an
// alias to another domain (e.g. a hosted checkout, status page, or
// support portal pointed at a vendor's own domain)
CNAME string `json:"cname,omitempty"`
// DKIMSelectors lists DKIM selector labels (see commonDKIMSelectors)
// found configured on this subdomain, e.g. "resend" for
// "resend._domainkey.mail.example.com"
DKIMSelectors []string `json:"dkim_selectors,omitempty"`
}
SubdomainDNSInfo holds the DNS records found for a single conventional subdomain probed alongside the apex
type System ¶
type System struct {
Name string `json:"name,omitempty"`
// Summary is a brief, 1-2 sentence description of the system
Summary string `json:"summary,omitempty"`
// FullDescription is a more thorough description of the system, drawn
// from documentation, architecture pages, or other technical content when available
FullDescription string `json:"full_description,omitempty"`
}
System describes a single technical system or component that makes up the company's offering (e.g. a web console, public API, mobile app, or storage backend)
type SystemEntry ¶
type SystemEntry struct {
// SystemName is the system's name
SystemName string `json:"system_name"`
// Description is the system's description
Description string `json:"description"`
}
SystemEntry is one of the company's systems, using field names that mirror Openlane's SystemDetail object
type Technology ¶
type Technology struct {
// Name is the technology's display name
Name string `json:"name"`
// URL is the technology's site or product URL, if known
URL string `json:"url,omitempty"`
// Categories are the technology's detected categories
Categories []string `json:"categories,omitempty"`
}
Technology is one detected technology entry in a scan report
type TransportSecurity ¶
type TransportSecurity struct {
// HTTPSReachable reports whether the domain answered over HTTPS
HTTPSReachable bool `json:"https_reachable"`
// RedirectsToHTTPS reports whether a plain HTTP request ended up on HTTPS
RedirectsToHTTPS bool `json:"redirects_to_https"`
// HSTS reports whether a Strict-Transport-Security header was sent over HTTPS
HSTS bool `json:"hsts"`
// HSTSMaxAge is the header's max-age directive in seconds
HSTSMaxAge int64 `json:"hsts_max_age,omitempty"`
// HSTSIncludeSubdomains reports whether the header carried includeSubDomains
HSTSIncludeSubdomains bool `json:"hsts_include_subdomains,omitempty"`
// HSTSPreload reports whether the header carried the preload directive
HSTSPreload bool `json:"hsts_preload,omitempty"`
}
TransportSecurity is the result of probing how strictly a domain enforces HTTPS
type TrustCenterPage ¶
type TrustCenterPage struct {
// HostedBy is the platform or vendor hosting this trust center (e.g.,
// Vanta, Drata, SafeBase, Whistic, Conveyor, TrustArc, OneTrust), or
// "self-hosted" if it appears to be a custom-built page
HostedBy string `json:"hosted_by,omitempty"`
// Frameworks lists compliance frameworks or certifications listed
Frameworks []string `json:"frameworks,omitempty"`
// SOC2Certified indicates whether the trust center claims SOC 2 (Type I or Type II) certification or compliance
SOC2Certified bool `json:"soc2_certified,omitempty"`
// Controls lists individual security or compliance controls or practices listed
Controls []string `json:"controls,omitempty"`
// Documents lists compliance documents or reports listed, each flagged public or gated
Documents []TrustDocument `json:"documents,omitempty"`
// Subprocessors lists third-party vendors or sub-processors mentioned
Subprocessors []string `json:"subprocessors,omitempty"`
}
TrustCenterPage holds structured information extracted from a trust center or trust portal page
type TrustDocument ¶
type TrustDocument struct {
// Name is the document title (e.g., "SOC 2 Type II Report")
Name string `json:"name,omitempty"`
// URL is a direct link to the document, if one is available without requesting access
URL string `json:"url,omitempty"`
// Public indicates the document can be viewed or downloaded without requesting access or signing an NDA
Public bool `json:"public,omitempty"`
}
TrustDocument is a single compliance document or report listed on a trust center
type Vendor ¶
type Vendor struct {
// Name is the vendor's canonical display name
Name string `json:"name"`
// LegalName is the vendor's raw, un-canonicalized legal entity name (e.g. "Cloudflare, Inc"),
// set only when it differs from Name
LegalName string `json:"legal_name,omitempty"`
// URL is the vendor's site or product URL, if known
URL string `json:"url,omitempty"`
// Categories are the vendor's detected categories
Categories []string `json:"categories,omitempty"`
}
Vendor is one detected vendor entry in a scan report
type WellKnown ¶
type WellKnown struct {
// SecurityTxt is the RFC 9116 security contact probe
SecurityTxt *SecurityTxt `json:"security_txt,omitempty"`
// Robots is the robots.txt AI crawler probe
Robots *RobotsTxt `json:"robots_txt,omitempty"`
// LLMs is the llms.txt probe
LLMs *LLMsTxt `json:"llms_txt,omitempty"`
// Transport is the HTTPS and HSTS enforcement probe
Transport *TransportSecurity `json:"transport,omitempty"`
}
WellKnown holds the well-known file and transport security probes for a domain. Every field is best-effort: a nil section means that probe found nothing or failed, which for scoring purposes is treated the same as absent
func GetWellKnown ¶
GetWellKnown probes domain for security.txt, robots.txt, llms.txt, and HTTPS enforcement, concurrently. Each probe is best-effort and independent, so a failure in one leaves the others intact; the returned error is non-nil only when domain itself could not be resolved to a registrable name