crawler

package
v1.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 14, 2024 License: GPL-3.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateColly

func CreateColly(delayTime int, concurrency int, cache bool, timeout int,
	intensive bool, rua bool, proxy string, userAgent string, target string) *colly.Collector

CreateColly takes as input all the settings needed to instantiate a new Colly Collector object and it returns this object.

func EndpointsMatch

func EndpointsMatch(target string, endpointsFile *[]string) []scanner.EndpointMatched

EndpointsMatch check if an endpoint matches a juicy parameter.

func ErrorsMatch

func ErrorsMatch(url, body string) []scanner.ErrorMatched

ErrorsMatch checks the patterns for errors.

func GenerateRandomUserAgent

func GenerateRandomUserAgent() string

GenerateRandomUserAgent generates a random user agent (can be Chrome or Firefox).

func GetRequest

func GetRequest(target string) (string, error)

GetRequest performs a GET request and return a string (the body of the response).

func HeadRequest

func HeadRequest(target string) (string, error)

HeadRequest performs a HEAD request and return a string (the headers of the response).

func IgnoreMatch

func IgnoreMatch(url string, ignoreSlice *[]string) bool

IgnoreMatch checks if the URL should be ignored or not.

func InfoMatch

func InfoMatch(url, body string) []scanner.InfoMatched

InfoMatch checks the patterns for infos.

func PostRequest

func PostRequest(target string, data map[string]string) (string, error)

PostRequest performs a POST request and return a string (the body of the response) the map in the input should contains the data fields and values in this way for example: { email: test@example.com, password: stupid_pwd }.

func RetrieveBody

func RetrieveBody(target *string) string

RetrieveBody retrieves the body (in the response) of a url.

func SecretsMatch

func SecretsMatch(url, body string, secretsFile *[]string) []scanner.SecretMatched

SecretsMatch checks if a body matches some secrets.

Types

type Event added in v1.3.0

type Event struct {
	ProtocolTemp string
	TargetTemp   string
	Target       string
	Intensive    bool
	Ignore       bool
	Debug        bool
	JSON         bool
	IgnoreSlice  []string
	URLs         *[]string
}

type Results added in v1.3.0

type Results struct {
	URLs       []string
	Secrets    []scanner.SecretMatched
	Endpoints  []scanner.EndpointMatched
	Extensions []scanner.FileTypeMatched
	Errors     []scanner.ErrorMatched
	Infos      []scanner.InfoMatched
}

func New

func New(scan *Scan) *Results

New it's the actual crawler engine. It controls all the behaviours of a scan (event handlers, secrets, errors, extensions and endpoints scanning).

type Scan added in v1.3.0

type Scan struct {
	// Flags
	Cache         bool
	Debug         bool
	EndpointsFlag bool
	ErrorsFlag    bool
	InfoFlag      bool
	Intensive     bool
	Plain         bool
	Rua           bool
	SecretsFlag   bool
	Ignore        string
	IgnoreTxt     string
	JSON          bool
	HTML          string
	Proxy         string
	Target        string
	Txt           string
	UserAgent     string
	FileType      int
	Headers       map[string]string
	StoreResp     bool

	// Settings
	Concurrency int
	Delay       int
	Timeout     int

	// Storage
	SecretsSlice   []string
	EndpointsSlice []string
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL