Documentation
¶
Overview ¶
Package remotesource loads parser-kit data from an HTTP endpoint.
It lives in its own package so that importing the root package does not drag http-kit into binaries that never fetch anything over HTTP. A service whose rules live on disk or in Redis pays nothing for remote support existing; only importing this package links it in.
Since http-kit v2 that is all it drags: OpenTelemetry, go-logr and golang.org/x/sys are gone from the graph. Trace propagation is now something a caller asks for -- see WithPropagator -- rather than something every importer links whether or not it traces.
rules, err := remotesource.New("https://config.internal/rules.json",
remotesource.WithAuthorization("Bearer "+token),
remotesource.WithTimeout(3*time.Second),
)
Security ¶
The URL is used as given and the Authorization header is sent with it. Nothing here validates the destination beyond its syntax, so passing a caller-controlled URL makes this an SSRF primitive that forwards credentials. Validate untrusted URLs before calling -- cli-kit's validator.ValidateURL plus validator.SSRFDialControl on the transport covers both the name and the address it actually resolves to at connect time, which a syntax check cannot.
Example ¶
A remote source with a local file behind it: if the endpoint is down, the file on disk answers.
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"time"
parserkit "github.com/soulteary/parser-kit/v3"
"github.com/soulteary/parser-kit/v3/remotesource"
)
type Rule struct {
ID string `json:"id"`
Action string `json:"action"`
}
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`[{"id":"r1","action":"allow"}]`))
}))
defer srv.Close()
remote, err := remotesource.New(srv.URL+"/rules.json",
remotesource.WithAuthorization("Bearer token"),
remotesource.WithTimeout(3*time.Second),
)
if err != nil {
panic(err)
}
loader, err := parserkit.NewLoader[Rule](nil)
if err != nil {
panic(err)
}
rules, err := loader.Load(context.Background(),
parserkit.At(0, remote),
parserkit.At(1, parserkit.File("/etc/app/rules.json").AllowMissing()),
)
if err != nil {
panic(err)
}
fmt.Println(rules[0].ID, rules[0].Action)
}
Output: r1 allow
Index ¶
- Constants
- type Fetcher
- type Option
- func WithAuthorization(value string) Option
- func WithClient(c *httpkit.Client) Option
- func WithHeader(name, value string) Option
- func WithInsecureSkipVerify() Option
- func WithPropagator(p httpkit.Propagator) Option
- func WithRetry(p RetryPolicy) Option
- func WithTimeout(d time.Duration) Option
- func WithUserAgent(ua string) Option
- type RetryPolicy
Examples ¶
Constants ¶
const DefaultTimeout = 5 * time.Second
DefaultTimeout bounds a single request when WithTimeout is not used.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher reads a JSON payload from one HTTP endpoint. It implements parserkit.Fetcher.
func New ¶
New returns a Fetcher reading rawURL.
It fails on a URL that is not an absolute http or https URL, and on a client it cannot build -- an unreadable CA file, say, when one was configured on a client passed to WithClient. A syntax check is not an SSRF defence; see the package documentation.
func (*Fetcher) Retry ¶
func (f *Fetcher) Retry() RetryPolicy
Retry returns the effective retry policy, after the defaults New filled in.
type Option ¶
type Option func(*settings)
Option configures a Fetcher.
func WithAuthorization ¶
WithAuthorization sets the Authorization header sent with the request. The value is used verbatim, so include the scheme: "Bearer "+token.
func WithClient ¶
WithClient hands the source an http-kit client to use instead of building one, so several remote sources can share one connection pool -- and so an application that already configures mTLS, a proxy or a custom transport can pass that configuration in.
The client's own timeout applies to the transport; WithTimeout still bounds the call through the request context.
func WithHeader ¶
WithHeader sets an extra request header. It may be used more than once.
func WithInsecureSkipVerify ¶
func WithInsecureSkipVerify() Option
WithInsecureSkipVerify disables TLS certificate verification. Only use it in development. It is ignored when WithClient supplies the client, which carries its own TLS configuration.
func WithPropagator ¶
func WithPropagator(p httpkit.Propagator) Option
WithPropagator injects cross-process context -- trace headers, baggage, a request ID -- into every request this source sends. There is none by default.
Until http-kit v2 this package called otel.GetTextMapPropagator() on every fetch, so a service that configured a global OpenTelemetry propagator got trace headers without asking, and every service that imported this package linked OpenTelemetry whether it traced or not. To get that behaviour back, ask for it:
import "github.com/soulteary/http-kit/v2/otelprop" remotesource.New(url, remotesource.WithPropagator(otelprop.Global()))
It applies to the client this package builds. A caller supplying its own through WithClient sets Propagator on that client's httpkit.Options instead, and this option is then ignored.
func WithRetry ¶
func WithRetry(p RetryPolicy) Option
WithRetry sets the retry policy. Non-positive fields take their defaults -- see RetryPolicy.
func WithTimeout ¶
WithTimeout bounds one request, retries included (default: DefaultTimeout). Zero or less leaves the request bounded only by the caller's context.
func WithUserAgent ¶
WithUserAgent sets the User-Agent header for requests this source makes. It is ignored when WithClient supplies the client.
type RetryPolicy ¶
type RetryPolicy struct {
// MaxRetries is how many times a failed request is retried (default: 3).
MaxRetries int
// RetryDelay is the delay before the first retry (default: 1s). Each
// subsequent delay doubles, up to MaxRetryDelay.
//
// It must be positive. Zero does not mean "retry once immediately" -- it
// means every retry fires immediately, which turns a failing remote
// source into a tight request loop, so New replaces a non-positive value
// with the default.
RetryDelay time.Duration
// MaxRetryDelay caps the backoff between retries (default: 30s).
//
// It must be positive for the same reason, and more sharply: http-kit's
// CalculateRetryDelay clamps every computed delay to this ceiling
// unconditionally, so a zero here removes the backoff however large
// RetryDelay is. New replaces a non-positive value with the default.
MaxRetryDelay time.Duration
}
RetryPolicy is the retry behaviour for one remote source.
In v1 these were fields on LoadOptions, which meant every source in a loader shared one policy whether or not any of them were remote.
Example ¶
Retries back off. A non-positive field is replaced by its default rather than honoured, because zero means "retry immediately, forever" -- see RetryPolicy.
package main
import (
"fmt"
"github.com/soulteary/parser-kit/v3/remotesource"
)
func main() {
f, err := remotesource.New("https://config.internal/rules.json",
remotesource.WithRetry(remotesource.RetryPolicy{MaxRetries: 5}),
)
if err != nil {
panic(err)
}
p := f.Retry()
fmt.Println(p.MaxRetries, p.RetryDelay, p.MaxRetryDelay)
}
Output: 5 1s 30s
func DefaultRetryPolicy ¶
func DefaultRetryPolicy() RetryPolicy
DefaultRetryPolicy returns the retry policy used when WithRetry is not.