remotesource

package
v3.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package remotesource loads parser-kit data from an HTTP endpoint.

It lives in its own package so that importing the root package does not drag http-kit into binaries that never fetch anything over HTTP. A service whose rules live on disk or in Redis pays nothing for remote support existing; only importing this package links it in.

Since http-kit v2 that is all it drags: OpenTelemetry, go-logr and golang.org/x/sys are gone from the graph. Trace propagation is now something a caller asks for -- see WithPropagator -- rather than something every importer links whether or not it traces.

rules, err := remotesource.New("https://config.internal/rules.json",
	remotesource.WithAuthorization("Bearer "+token),
	remotesource.WithTimeout(3*time.Second),
)

Security

The URL is used as given and the Authorization header is sent with it. Nothing here validates the destination beyond its syntax, so passing a caller-controlled URL makes this an SSRF primitive that forwards credentials. Validate untrusted URLs before calling -- cli-kit's validator.ValidateURL plus validator.SSRFDialControl on the transport covers both the name and the address it actually resolves to at connect time, which a syntax check cannot.

Example

A remote source with a local file behind it: if the endpoint is down, the file on disk answers.

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	parserkit "github.com/soulteary/parser-kit/v3"
	"github.com/soulteary/parser-kit/v3/remotesource"
)

type Rule struct {
	ID     string `json:"id"`
	Action string `json:"action"`
}

func main() {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		_, _ = w.Write([]byte(`[{"id":"r1","action":"allow"}]`))
	}))
	defer srv.Close()

	remote, err := remotesource.New(srv.URL+"/rules.json",
		remotesource.WithAuthorization("Bearer token"),
		remotesource.WithTimeout(3*time.Second),
	)
	if err != nil {
		panic(err)
	}

	loader, err := parserkit.NewLoader[Rule](nil)
	if err != nil {
		panic(err)
	}

	rules, err := loader.Load(context.Background(),
		parserkit.At(0, remote),
		parserkit.At(1, parserkit.File("/etc/app/rules.json").AllowMissing()),
	)
	if err != nil {
		panic(err)
	}

	fmt.Println(rules[0].ID, rules[0].Action)
}
Output:
r1 allow

Index

Examples

Constants

View Source
const DefaultTimeout = 5 * time.Second

DefaultTimeout bounds a single request when WithTimeout is not used.

Variables

This section is empty.

Functions

This section is empty.

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher reads a JSON payload from one HTTP endpoint. It implements parserkit.Fetcher.

func New

func New(rawURL string, opts ...Option) (*Fetcher, error)

New returns a Fetcher reading rawURL.

It fails on a URL that is not an absolute http or https URL, and on a client it cannot build -- an unreadable CA file, say, when one was configured on a client passed to WithClient. A syntax check is not an SSRF defence; see the package documentation.

func (*Fetcher) Fetch

func (f *Fetcher) Fetch(ctx context.Context, maxBytes int64) ([]byte, error)

Fetch performs the request, subject to maxBytes.

func (*Fetcher) Retry

func (f *Fetcher) Retry() RetryPolicy

Retry returns the effective retry policy, after the defaults New filled in.

func (*Fetcher) URL

func (f *Fetcher) URL() string

URL returns the endpoint this fetcher reads.

type Option

type Option func(*settings)

Option configures a Fetcher.

func WithAuthorization

func WithAuthorization(value string) Option

WithAuthorization sets the Authorization header sent with the request. The value is used verbatim, so include the scheme: "Bearer "+token.

func WithClient

func WithClient(c *httpkit.Client) Option

WithClient hands the source an http-kit client to use instead of building one, so several remote sources can share one connection pool -- and so an application that already configures mTLS, a proxy or a custom transport can pass that configuration in.

The client's own timeout applies to the transport; WithTimeout still bounds the call through the request context.

func WithHeader

func WithHeader(name, value string) Option

WithHeader sets an extra request header. It may be used more than once.

func WithInsecureSkipVerify

func WithInsecureSkipVerify() Option

WithInsecureSkipVerify disables TLS certificate verification. Only use it in development. It is ignored when WithClient supplies the client, which carries its own TLS configuration.

func WithPropagator

func WithPropagator(p httpkit.Propagator) Option

WithPropagator injects cross-process context -- trace headers, baggage, a request ID -- into every request this source sends. There is none by default.

Until http-kit v2 this package called otel.GetTextMapPropagator() on every fetch, so a service that configured a global OpenTelemetry propagator got trace headers without asking, and every service that imported this package linked OpenTelemetry whether it traced or not. To get that behaviour back, ask for it:

import "github.com/soulteary/http-kit/v2/otelprop"

remotesource.New(url, remotesource.WithPropagator(otelprop.Global()))

It applies to the client this package builds. A caller supplying its own through WithClient sets Propagator on that client's httpkit.Options instead, and this option is then ignored.

func WithRetry

func WithRetry(p RetryPolicy) Option

WithRetry sets the retry policy. Non-positive fields take their defaults -- see RetryPolicy.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout bounds one request, retries included (default: DefaultTimeout). Zero or less leaves the request bounded only by the caller's context.

func WithUserAgent

func WithUserAgent(ua string) Option

WithUserAgent sets the User-Agent header for requests this source makes. It is ignored when WithClient supplies the client.

type RetryPolicy

type RetryPolicy struct {
	// MaxRetries is how many times a failed request is retried (default: 3).
	MaxRetries int

	// RetryDelay is the delay before the first retry (default: 1s). Each
	// subsequent delay doubles, up to MaxRetryDelay.
	//
	// It must be positive. Zero does not mean "retry once immediately" -- it
	// means every retry fires immediately, which turns a failing remote
	// source into a tight request loop, so New replaces a non-positive value
	// with the default.
	RetryDelay time.Duration

	// MaxRetryDelay caps the backoff between retries (default: 30s).
	//
	// It must be positive for the same reason, and more sharply: http-kit's
	// CalculateRetryDelay clamps every computed delay to this ceiling
	// unconditionally, so a zero here removes the backoff however large
	// RetryDelay is. New replaces a non-positive value with the default.
	MaxRetryDelay time.Duration
}

RetryPolicy is the retry behaviour for one remote source.

In v1 these were fields on LoadOptions, which meant every source in a loader shared one policy whether or not any of them were remote.

Example

Retries back off. A non-positive field is replaced by its default rather than honoured, because zero means "retry immediately, forever" -- see RetryPolicy.

package main

import (
	"fmt"

	"github.com/soulteary/parser-kit/v3/remotesource"
)

func main() {
	f, err := remotesource.New("https://config.internal/rules.json",
		remotesource.WithRetry(remotesource.RetryPolicy{MaxRetries: 5}),
	)
	if err != nil {
		panic(err)
	}

	p := f.Retry()
	fmt.Println(p.MaxRetries, p.RetryDelay, p.MaxRetryDelay)
}
Output:
5 1s 30s

func DefaultRetryPolicy

func DefaultRetryPolicy() RetryPolicy

DefaultRetryPolicy returns the retry policy used when WithRetry is not.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL