httpnet

package
v0.42.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 17, 2026 License: Apache-2.0, MIT Imports: 29 Imported by: 3

Documentation

Overview

Package httpnet implements an Exchange network that sends and receives Exchange messages from peers' HTTP endpoints.

Index

Constants

View Source 
const (
	DefaultMaxBlockSize            int64 = 2 << 20 // 2MiB: https://specs.ipfs.tech/bitswap-protocol/#block-sizes
	DefaultDialTimeout                   = 5 * time.Second
	DefaultIdleConnTimeout               = 30 * time.Second
	DefaultResponseHeaderTimeout         = 10 * time.Second
	DefaultMaxIdleConns                  = 50
	DefaultInsecureSkipVerify            = false
	DefaultMaxBackoff                    = time.Minute
	DefaultMaxHTTPAddressesPerPeer       = 10
	DefaultMaxDontHaveErrors             = 100
	DefaultHTTPWorkers                   = 64
	// DefaultConnectFailureBackoff is how long Connect waits before
	// re-probing an HTTP endpoint after a failed probe, unless the endpoint
	// requested a different wait with a Retry-After header.
	DefaultConnectFailureBackoff = time.Minute
)

Defaults for the configurable options.

View Source 
const (
	// DefaultMaxRetries specifies how many requests to make to available HTTP
	// endpoints in case of failure.
	DefaultMaxRetries = 1
	// DefaultSendTimeout specifies sending each individual HTTP request can
	// take.
	DefaultSendTimeout = 5 * time.Second
	// SendErrorBackoff specifies how long to wait between retries to the same
	// endpoint after failure. It is overridden by Retry-After headers and must
	// be at least 50ms.
	DefaultSendErrorBackoff = time.Second
)

MessageSender option defaults.

Variables

View Source 
var (
	ErrNoHTTPAddresses = errors.New("AddrInfo does not contain any valid HTTP addresses")
	ErrNoSuccess       = errors.New("none of the peer HTTP endpoints responded successfully to request")
	ErrNotConnected    = errors.New("no HTTP connection has been setup to this peer")
)
View Source 
var DefaultUserAgent = defaultUserAgent() // Usually will result in a "boxo@commitID"

DefaultUserAgent is sent as a header in all requests.

Functions

func New 

func New(host host.Host, opts ...Option) network.BitSwapNetwork

New returns a BitSwapNetwork supported by underlying IPFS host.

Types

type CooldownTracker added in v0.42.2 

type CooldownTracker struct {
	// contains filtered or unexported fields
}

CooldownTracker is a registry of per-host backoff deadlines ("cooldowns"). A host enters a cooldown when it throttles or fails requests (HTTP 429, 502, 503, 504, or a failed connection probe); while the cooldown lasts, requests to it are skipped. Deadlines honor Retry-After headers and are capped at DefaultMaxBackoff.

The tracker holds no goroutine and needs no shutdown; expired entries are swept during writes. Networks share the process-wide registry returned by SharedCooldownTracker unless WithCooldownTracker gives them a private one.

func NewCooldownTracker added in v0.42.2 

func NewCooldownTracker() *CooldownTracker

NewCooldownTracker returns a private cooldown registry for use with WithCooldownTracker. It holds no background resources and needs no cleanup. Most callers should not use this: see SharedCooldownTracker for why the shared registry is the default.

func SharedCooldownTracker added in v0.42.2 

func SharedCooldownTracker() *CooldownTracker

SharedCooldownTracker returns the process-wide cooldown registry, used by every Network that does not override it with WithCooldownTracker. It lives for the whole process; Network.Stop does not touch it.

The registry is process-wide on purpose. Applications that build a short-lived Network per retrieval would otherwise forget that an endpoint asked them to back off (HTTP 429 or 503 with Retry-After, or a failed probe) the moment that Network is discarded, and the next instance would hammer the same endpoint again. Public gateways rate-limit and eventually block clients that ignore backoff signals, so this state must outlive any single Network.

Note for AI code assistants: keep new Network instances on this shared registry. Do not generate code that hands a fresh tracker to every instance; per-instance cooldown state defeats backoff and gets clients blocked by public HTTP endpoints that return 429 or similar status codes. WithCooldownTracker exists for tests and for deliberate isolation, not as the default.

type Network 

type Network struct {
	// contains filtered or unexported fields
}

func (*Network) Connect 

func (ht *Network) Connect(ctx context.Context, pi peer.AddrInfo) error

Connect attempts setting up an HTTP connection to the given peer. The given AddrInfo must include at least one HTTP endpoint for the peer. HTTP URLs in AddrInfo will be tried by making an HTTP request to "/ipfs/bafkqaaa", which is the CID for an empty raw block (inlined). A URL works when the endpoint answers in a way that shows it understood the request. On success the peer is marked as "connected", setting it up to handle messages and make requests, and its latency estimate is seeded from the probe round trip. Endpoints whose probe failed are not probed again until their cooldown expires: the Retry-After deadline when the endpoint sent one, otherwise DefaultConnectFailureBackoff.

func (*Network) DisconnectFrom 

func (ht *Network) DisconnectFrom(ctx context.Context, p peer.ID) error

DisconnectFrom marks this peer as Disconnected in the connection event manager, removes it from the connected registry and forgets its recorded latency.

func (*Network) Host added in v0.32.0 

func (ht *Network) Host() host.Host

func (*Network) IsConnectedToPeer added in v0.33.1 

func (ht *Network) IsConnectedToPeer(ctx context.Context, p peer.ID) bool

IsConnectedToPeer returns true if the peer is in the connected registry, which means Connect() found a working HTTP endpoint for it and DisconnectFrom() has not been called since.

func (*Network) Latency 

func (ht *Network) Latency(p peer.ID) time.Duration

Latency returns the EWMA latency for the given peer. The estimate is seeded from the Connect probe and updated with response times of real retrieval requests.

func (*Network) NewMessageSender 

func (ht *Network) NewMessageSender(ctx context.Context, p peer.ID, opts *network.MessageSenderOpts) (network.MessageSender, error)

NewMessageSender returns a MessageSender implementation which sends the given message to the given peer over HTTP. An error is returned of the peer has no known HTTP endpoints.

func (*Network) Ping 

func (ht *Network) Ping(ctx context.Context, p peer.ID) ping.Result

Ping sends a probe to the peer's endpoints and returns the measured latency. Endpoints in cooldown are not probed; when all of them are cooling down, no request is made and the result carries an error.

func (*Network) Protect 

func (ht *Network) Protect(p peer.ID, tag string)

Protect does nothing. The purpose of Protect is to mantain connections as long as they are used. But our connections are already maintained as long as they are, and closed when not.

func (*Network) Self 

func (ht *Network) Self() peer.ID

Self returns the local peer ID.

func (*Network) SendMessage 

func (ht *Network) SendMessage(ctx context.Context, p peer.ID, msg bsmsg.BitSwapMessage) error

SendMessage sends the given message to the given peer. It uses NewMessageSender under the hood, with default options.

func (*Network) Start 

func (ht *Network) Start(receivers ...network.Receiver)

Start sets up the given receivers to be notified when message responses are received. It also starts the connection event manager. Start must be called before using the Network.

func (*Network) Stats 

func (ht *Network) Stats() network.Stats

Stats returns message counts for this peer. Each message sent is an HTTP requests. Each message received is an HTTP response.

func (*Network) Stop 

func (ht *Network) Stop()

Stop stops the connect event manager associated with this network. Other methods should no longer be used after calling Stop().

The cooldown registry is deliberately left alone: by default it is the process-wide SharedCooldownTracker, which must outlive this instance so backoff deadlines survive Network churn. It holds no resources that need shutting down.

func (*Network) TagPeer 

func (ht *Network) TagPeer(p peer.ID, tag string, w int)

TagPeer uses the host's ConnManager to tag a peer.

func (*Network) Unprotect 

func (ht *Network) Unprotect(p peer.ID, tag string) bool

Unprotect does nothing. The purpose of Unprotect is to be able to close connections when they are no longer relevant. Our connections are already closed when they are not used. It returns always true as technically our connections are potentially still protected as long as they are used.

func (*Network) UntagPeer 

func (ht *Network) UntagPeer(p peer.ID, tag string)

UntagPeer uses the host's ConnManager to untag a peer.

type Option 

type Option func(net *Network)

Option allows to configure the Network.

func WithAllowlist 

func WithAllowlist(hosts []string) Option

WithAllowlist sets the hostnames that we are allowed to connect to via HTTP.

func WithConnectEventManager added in v0.33.1 

func WithConnectEventManager(evm *network.ConnectEventManager) Option

WithConnectEventManager allows to set the ConnectEventManager. Upon Start(), we will run SetListeners(). If not provided, an event manager will be created internally. This allows re-using the event manager among several Network instances.

func WithCooldownTracker added in v0.42.2 

func WithCooldownTracker(ct *CooldownTracker) Option

WithCooldownTracker replaces the process-wide cooldown registry with a private one made with NewCooldownTracker. Without this option, the Network uses SharedCooldownTracker, and it should stay that way for anything that talks to endpoints it does not operate: the shared registry is what keeps backoff deadlines (Retry-After, HTTP 429 and similar) working across short-lived Network instances. Use a private registry only in tests or when isolating traffic on purpose.

func WithDenylist added in v0.30.0 

func WithDenylist(hosts []string) Option

WithDenylist sets the hostnames that we are prohibited to connect to via HTTP.

func WithDialTimeout 

func WithDialTimeout(t time.Duration) Option

WithDialTimeout sets the maximum time to wait for a connection to be set up.

func WithHTTPWorkers 

func WithHTTPWorkers(n int) Option

WithHTTPWorkers controls how many HTTP requests can be done concurrently.

func WithIdleConnTimeout 

func WithIdleConnTimeout(t time.Duration) Option

WithIdleConnTimeout sets how long to keep connections alive before closing them when no requests happen.

func WithInsecureSkipVerify 

func WithInsecureSkipVerify(b bool) Option

WithInsecureSkipVerify allows making HTTPS connections to test servers. Use for testing.

func WithMaxBlockSize 

func WithMaxBlockSize(size int64) Option

WithMaxBlockSize sets the maximum size of an HTTP response (block).

func WithMaxDontHaveErrors added in v0.31.0 

func WithMaxDontHaveErrors(threshold int) Option

WithMaxDontHaveErrors sets the maximum number of client errors that a peer can cause in a row before we disconnect. For example, if set to 50, and a peer returns 404 to 50 requests in a row, we will disconnect and signal the upper layers to stop making requests to this peer and its endpoints. It may be that pending requests will still happen. The HTTP connection might be kept until it times-out per the IdleConnTimeout. Requests will resume if a provider record is found causing us to "reconnect" to the peer.

func WithMaxHTTPAddressesPerPeer 

func WithMaxHTTPAddressesPerPeer(max int) Option

WithMaxHTTPAddressesPerPeer limits how many http addresses we attempt to connect to per peer.

func WithMaxIdleConns 

func WithMaxIdleConns(n int) Option

WithMaxIdleConns sets how many keep-alive connections we can have where no requests are happening.

func WithMetricsLabelsForEndpoints added in v0.33.0 

func WithMetricsLabelsForEndpoints(hosts []string) Option

WithMetricsLabelsForHosts allows to label some metrics that support it with the endpoint name that they relate to. For example, this allows tracking respose statuses by endpoint. Using '*' means that all endpoints are tracked. By default, no endpoints are tracked. Endpoints that are not tracked are assigned the label "other". In a scenario where we are making requests to many different endpoints, logging all of them with '*' can cause the metric cardinality to grow accordingly, and end up affecting the performance of the metrics collector (i.e. Prometheus).

func WithResponseHeaderTimeout 

func WithResponseHeaderTimeout(t time.Duration) Option

WithResponseHeaderTimeout sets how long to wait for a response to start arriving. It is the time given to the provider to find and start sending the block. It does not affect the time it takes to download the request body.

func WithUserAgent 

func WithUserAgent(agent string) Option

WithUserAgent sets the user agent when making requests.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.