urlresolver

package module

v0.1.4 Latest Latest Go to latest Published: Apr 14, 2023 License: MIT Imports: 17 Imported by: 4

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/mccutchen/urlresolver

Links

Open Source Insights

README ¶

urlresolver

A golang package that "resolves" a given URL by issuing a GET request, following any redirects, canonicalizing the final URL, and attempting to extract the title from the final response body.

Methodology

Resolving

A URL is resolved by issuing a GET request and following any redirects until a non-30x response is received.

Canonicalizing

The final URL is aggressively canonicalized using a combination of PuerkitoBio/purell and some manual heuristics for removing unnecessary query params (e.g. utm_* tracking params), normalizing case (e.g. twitter.com/Thresholderbot and twitter.com/thresholderbot are the same).

Canonicalization is optimized for URLs that are shared on social media.

Security

TL;DR: Use safedialer.Control in the transport's dialer to block attempts to resolve URLs pointing at internal, private IP addresses.

Exposing functionality like this on the internet can be dangerous, because it could theoretically allow a malicious client to discover information about your internal network by asking it to resolve URLs pointing at private IP addresses.

The dangers, along with a golang-specific mitigation, are outlined in Andrew Ayer's excellent "Preventing Server Side Request Forgery in Golang" blog post.

To mitigate that danger, users are strongly encouraged to use safedialer.Control as the Control function in the dialer used by the transport given to urlresolver.New.

Documentation ¶

Index ¶

Variables
func Canonicalize(u *url.URL) string
type Interface
type Resolver
- func New(transport http.RoundTripper, timeout time.Duration) *Resolver
- func (r *Resolver) Resolve(ctx context.Context, givenURL string) (Result, error)
type Result

Constants ¶

This section is empty.

Variables ¶

View Source

var NormalizationFlags = (purell.FlagsSafe |
	purell.FlagRemoveDotSegments |
	purell.FlagRemoveDuplicateSlashes |
	purell.FlagDecodeDWORDHost |
	purell.FlagDecodeOctalHost |
	purell.FlagDecodeHexHost |
	purell.FlagRemoveUnnecessaryHostDots |
	purell.FlagRemoveEmptyPortSeparator)

NormalizationFlags defines the normalization flags the purell package will use during canonicalization.

See https://godoc.org/github.com/PuerkitoBio/purell#NormalizationFlags

Functions ¶

func Canonicalize ¶

func Canonicalize(u *url.URL) string

Canonicalize filters unnecessary query params and then normalizes a URL, ensuring consistent case, encoding, sorting of params, etc.

Types ¶

type Interface ¶

type Interface interface {
	Resolve(context.Context, string) (Result, error)
}

Interface defines the interface for a URL resolver.

type Resolver ¶

type Resolver struct {
	// contains filtered or unexported fields
}

Resolver resolves URLs.

func New ¶

func New(transport http.RoundTripper, timeout time.Duration) *Resolver

New creates a new Resolver that uses the given transport to make HTTP requests and applies the given timeout to the overall process (including any redirects that must be followed).

func (*Resolver) Resolve ¶

func (r *Resolver) Resolve(ctx context.Context, givenURL string) (Result, error)

Resolve resolves the given URL by following any redirects, canonicalizing the final URL, and attempting to extract the title from the final response body.

type Result ¶

type Result struct {
	ResolvedURL      string
	Title            string
	IntermediateURLs []string
}

Result is the result of resolving a URL.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
bufferpool
fakebrowser
safedialer Package safedialer provides a net.Dialer Control function that permits only TCP connections to port 80 and 443 on public IP addresses, so that an application may safely connect to possibly-malicious URLs controlled by external clients.	Package safedialer provides a net.Dialer Control function that permits only TCP connections to port 80 and 443 on public IP addresses, so that an application may safely connect to possibly-malicious URLs controlled by external clients.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL