crawler

package
v1.4.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2025 License: GPL-3.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateColly

func CreateColly(delayTime, concurrency, timeout, maxDepth int,
	cache, intensive, rua bool,
	proxy string, userAgent string, target string) *colly.Collector

CreateColly takes as input all the settings needed to instantiate a new Colly Collector object and it returns this object.

func EndpointsMatch

func EndpointsMatch(target string, endpointsFile *[]string) []scanner.EndpointMatched

EndpointsMatch check if an endpoint matches a juicy parameter.

func GenerateRandomUserAgent

func GenerateRandomUserAgent() string

GenerateRandomUserAgent generates a random user agent (can be Chrome or Firefox).

func GetRequest

func GetRequest(target string) (string, error)

GetRequest performs a GET request and return a string (the body of the response).

func HeadRequest

func HeadRequest(target string) (string, error)

HeadRequest performs a HEAD request and return a string (the headers of the response).

func IgnoreMatch

func IgnoreMatch(url string, ignoreSlice *[]string) bool

IgnoreMatch checks if the URL should be ignored or not.

func PostRequest

func PostRequest(target string, data map[string]string) (string, error)

PostRequest performs a POST request and return a string (the body of the response) the map in the input should contains the data fields and values in this way for example: { email: test@example.com, password: stupid_pwd }.

func RetrieveBody

func RetrieveBody(target *string) string

RetrieveBody retrieves the body (in the response) of a url.

Types

type Event added in v1.3.0

type Event struct {
	ProtocolTemp string
	TargetTemp   string
	Target       string
	Intensive    bool
	Ignore       bool
	Debug        bool
	JSON         bool
	IgnoreSlice  []string
	URLs         *[]string
}

type Results added in v1.3.0

type Results struct {
	URLs       []string
	Secrets    []scanner.SecretMatched
	Endpoints  []scanner.EndpointMatched
	Extensions []scanner.FileTypeMatched
	Errors     []scanner.ErrorMatched
	Infos      []scanner.InfoMatched
}

func New

func New(scan *Scan) *Results

New it's the actual crawler engine. It controls all the behaviours of a scan (event handlers, secrets, errors, extensions and endpoints scanning).

type Scan added in v1.3.0

type Scan struct {
	// Flags
	Cache            bool
	Debug            bool
	EndpointsFlag    bool
	ErrorsFlag       bool
	InfoFlag         bool
	Intensive        bool
	Plain            bool
	Rua              bool
	SecretsFlag      bool
	Ignore           string
	IgnoreTxt        string
	JSON             bool
	HTML             string
	Proxy            string
	Target           string
	Txt              string
	UserAgent        string
	FileType         int
	Headers          map[string]string
	StoreResp        bool
	MaxDepth         int
	IgnoreExtensions input.StringSlice

	// Settings
	Concurrency int
	Delay       int
	Timeout     int

	// Storage
	SecretsSlice   []string
	EndpointsSlice []string
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL