webanalyze

package module
v0.0.0-...-b70ccdb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 7, 2023 License: MIT Imports: 13 Imported by: 0

README

webanalyze

This is a port of Wappalyzer in Go. This tool is designed to be performant and allows to test huge lists of hosts.

Installation and usage

Precompiled releases

Precompiled releases can be downloaded directly here.

Build

If you want to build for yourself:

$ go install -v github.com/rverton/webanalyze/cmd/webanalyze@latest
$ webanalyze -update # loads new technologies.json file from wappalyzer project
$ webanalyze -h
Usage of webanalyze:
  -apps string
        app definition file. (default "technologies.json")
  -crawl int
        links to follow from the root page (default 0)
  -host string
        single host to test
  -hosts string
        filename with hosts, one host per line.
  -output string
        output format (stdout|csv|json) (default "stdout")
  -search
        searches all urls with same base domain (i.e. example.com and sub.example.com) (default true)
  -silent
	    avoid printing header (default false)
  -update
        update apps file
  -worker int
        number of worker (default 4)

The -update flags downloads a current version of technologies.json from the wappalyzer repository to the current folder.

Docker
# Clone the repo
git clone https://github.com/rverton/webanalyze.git
# Build the container
docker build -t webanalyze:latest webanalyze
# Run the container
docker run -it webanalyze:latest -h

Development / Usage as a lib

See cmd/webanalyze/main.go for an example on how to use this as a library.

Example

$ ./webanalyze -host robinverton.de -crawl 1
 :: webanalyze        : v1.0
 :: workers           : 4
 :: apps              : technologies.json
 :: crawl count       : 1
 :: search subdomains : true

https://robinverton.de/hire/ (0.5s):
    Highlight.js,  (Miscellaneous)
    Netlify,  (Web Servers, CDN)
    Google Font API,  (Font Scripts)
http://robinverton.de (0.8s):
    Highlight.js,  (Miscellaneous)
    Netlify,  (Web Servers, CDN)
    Hugo, 0.42.1 (Static Site Generator)
    Google Font API,  (Font Scripts)

$ ./webanalyze -host robinverton.de -crawl 1 -output csv
 :: webanalyze        : v1.0
 :: workers           : 4
 :: apps              : technologies.json
 :: crawl count       : 1
 :: search subdomains : true

Host,Category,App,Version
https://robinverton.de/hire/,Miscellaneous,Highlight.js,
https://robinverton.de/hire/,Font Scripts,Google Font API,
https://robinverton.de/hire/,"Web Servers,CDN",Netlify,
http://robinverton.de,"Web Servers,CDN",Netlify,
http://robinverton.de,Static Site Generator,Hugo,0.42.1
http://robinverton.de,Miscellaneous,Highlight.js,
http://robinverton.de,Font Scripts,Google Font API,

Documentation

Index

Constants

View Source 
const VERSION = "0.3.7-sk"
View Source 
const WappazlyerRoot = "https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src"

Variables

This section is empty.

Functions

func DownloadFile 

func DownloadFile(to string) error

DownloadFile pulls the latest technologies.json file from the Wappalyzer github

Types

type App 

type App struct {
	Cats     StringArray            `json:"cats"`
	CatNames []string               `json:"category_names"`
	Cookies  map[string]string      `json:"cookies"`
	Headers  map[string]string      `json:"headers"`
	Meta     map[string]StringArray `json:"meta"`
	HTML     StringArray            `json:"html"`
	Script   StringArray            `json:"scripts"`
	URL      StringArray            `json:"url"`
	Website  string                 `json:"website"`
	Implies  StringArray            `json:"implies"`

	HTMLRegex   []AppRegexp `json:"-"`
	ScriptRegex []AppRegexp `json:"-"`
	URLRegex    []AppRegexp `json:"-"`
	HeaderRegex []AppRegexp `json:"-"`
	MetaRegex   []AppRegexp `json:"-"`
	CookieRegex []AppRegexp `json:"-"`
}

App type encapsulates all the data about an App from technologies.json

func (*App) FindInHeaders 

func (app *App) FindInHeaders(headers http.Header) (matches [][]string, version string)

type AppRegexp 

type AppRegexp struct {
	Name    string
	Regexp  *regexp.Regexp
	Version string
}

type AppsDefinition 

type AppsDefinition struct {
	Apps map[string]App      `json:"technologies"`
	Cats map[string]Category `json:"categories"`
}

AppsDefinition type encapsulates the json encoding of the whole technologies.json file

type Category 

type Category struct {
	Name string `json:"name"`
}

Category names defined by wappalyzer

type Job 

type Job struct {
	URL             string
	Body            []byte
	Headers         http.Header //map[string][]string
	Cookies         []*http.Cookie
	Crawl           int
	SearchSubdomain bool
	// contains filtered or unexported fields
}

Job may consist only of a URL, in which case webanalyse will proceed to download from that URL, or it may consist of the Body and Headers of a request to a URL and the URL itself, in which case these fields will be trusted and used for analysis without further network traffic. If a Job is constructed using the OfflineJob constructor then a flag will be set to prevent downloading regardless of the contents (or absence) of the Body or Headers fields.

func NewOfflineJob 

func NewOfflineJob(url, body string, headers map[string][]string) *Job

NewOfflineJob constructs a job out of the constituents of a webanalyzer analysis; a URL, a body, and response headers. This constructor also sets a flag to explicitly prevent fetching from the URL even if the body and headers are nil or empty. Use this for...offline jobs.

func NewOnlineJob 

func NewOnlineJob(url, body string, headers map[string][]string, crawlCount int, searchSubdomain bool, redirect bool) *Job

NewOnlineJob constructs a job that may either have a URL only, or a URL, Body and Headers. If it contains at least a URL and Body, then webanalyzer will not re-download the data, but if a Body is absent then downloading will be attempted.

type Match 

type Match struct {
	App     `json:"app"`
	AppName string     `json:"app_name"`
	Matches [][]string `json:"matches"`
	Version string     `json:"version"`
}

Match type encapsulates the App information from a match on a document

type Result 

type Result struct {
	Host      string        `json:"host"`
	Matches   []Match       `json:"matches"`
	Duration  time.Duration `json:"duration"`
	Error     error         `json:"error"`
	Timestamp time.Time     `json:"timestamp"`
}

Result type encapsulates the result information from a given host

type StringArray 

type StringArray []string

StringArray type is a wrapper for []string for use in unmarshalling the technologies.json

func (*StringArray) UnmarshalJSON 

func (t *StringArray) UnmarshalJSON(data []byte) error

UnmarshalJSON is a custom unmarshaler for handling bogus technologies.json types from wappalyzer

type WebAnalyzer 

type WebAnalyzer struct {
	// contains filtered or unexported fields
}

WebAnalyzer types holds an analyzation job

func NewWebAnalyzer 

func NewWebAnalyzer(apps io.Reader, client *http.Client) (*WebAnalyzer, error)

NewWebAnalyzer initializes webanalyzer by passing a reader of the app definition and an schedulerChan, which allows the scanner to add scan jobs on its own

func (*WebAnalyzer) CategoryById 

func (wa *WebAnalyzer) CategoryById(cid string) string

func (*WebAnalyzer) Process 

func (wa *WebAnalyzer) Process(job *Job) (Result, []string)

worker loops until channel is closed. processes a single host at once

Source Files

View all Source files

Directories

Path Synopsis
cmd
webanalyze command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.