Documentation
¶
Overview ¶
Package remotesource loads parser-kit data from an HTTP endpoint.
It lives in its own package so that importing the root package does not drag http-kit -- and with it OpenTelemetry, go-logr and golang.org/x/sys -- into binaries that never fetch anything over HTTP. A service whose rules live on disk or in Redis pays nothing for remote support existing; only importing this package links it in.
rules, err := remotesource.New("https://config.internal/rules.json",
remotesource.WithAuthorization("Bearer "+token),
remotesource.WithTimeout(3*time.Second),
)
Security ¶
The URL is used as given and the Authorization header is sent with it. Nothing here validates the destination beyond its syntax, so passing a caller-controlled URL makes this an SSRF primitive that forwards credentials. Validate untrusted URLs before calling -- cli-kit's validator.ValidateURL plus validator.SSRFDialControl on the transport covers both the name and the address it actually resolves to at connect time, which a syntax check cannot.
Example ¶
A remote source with a local file behind it: if the endpoint is down, the file on disk answers.
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"time"
parserkit "github.com/soulteary/parser-kit/v2"
"github.com/soulteary/parser-kit/v2/remotesource"
)
type Rule struct {
ID string `json:"id"`
Action string `json:"action"`
}
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`[{"id":"r1","action":"allow"}]`))
}))
defer srv.Close()
remote, err := remotesource.New(srv.URL+"/rules.json",
remotesource.WithAuthorization("Bearer token"),
remotesource.WithTimeout(3*time.Second),
)
if err != nil {
panic(err)
}
loader, err := parserkit.NewLoader[Rule](nil)
if err != nil {
panic(err)
}
rules, err := loader.Load(context.Background(),
parserkit.At(0, remote),
parserkit.At(1, parserkit.File("/etc/app/rules.json").AllowMissing()),
)
if err != nil {
panic(err)
}
fmt.Println(rules[0].ID, rules[0].Action)
}
Output: r1 allow
Index ¶
Examples ¶
Constants ¶
const DefaultTimeout = 5 * time.Second
DefaultTimeout bounds a single request when WithTimeout is not used.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher reads a JSON payload from one HTTP endpoint. It implements parserkit.Fetcher.
func New ¶
New returns a Fetcher reading rawURL.
It fails on a URL that is not an absolute http or https URL, and on a client it cannot build -- an unreadable CA file, say, when one was configured on a client passed to WithClient. A syntax check is not an SSRF defence; see the package documentation.
func (*Fetcher) Retry ¶
func (f *Fetcher) Retry() RetryPolicy
Retry returns the effective retry policy, after the defaults New filled in.
type Option ¶
type Option func(*settings)
Option configures a Fetcher.
func WithAuthorization ¶
WithAuthorization sets the Authorization header sent with the request. The value is used verbatim, so include the scheme: "Bearer "+token.
func WithClient ¶
WithClient hands the source an http-kit client to use instead of building one, so several remote sources can share one connection pool -- and so an application that already configures mTLS, a proxy or a custom transport can pass that configuration in.
The client's own timeout applies to the transport; WithTimeout still bounds the call through the request context.
func WithHeader ¶
WithHeader sets an extra request header. It may be used more than once.
func WithInsecureSkipVerify ¶
func WithInsecureSkipVerify() Option
WithInsecureSkipVerify disables TLS certificate verification. Only use it in development. It is ignored when WithClient supplies the client, which carries its own TLS configuration.
func WithRetry ¶
func WithRetry(p RetryPolicy) Option
WithRetry sets the retry policy. Non-positive fields take their defaults -- see RetryPolicy.
func WithTimeout ¶
WithTimeout bounds one request, retries included (default: DefaultTimeout). Zero or less leaves the request bounded only by the caller's context.
func WithUserAgent ¶
WithUserAgent sets the User-Agent header for requests this source makes. It is ignored when WithClient supplies the client.
type RetryPolicy ¶
type RetryPolicy struct {
// MaxRetries is how many times a failed request is retried (default: 3).
MaxRetries int
// RetryDelay is the delay before the first retry (default: 1s). Each
// subsequent delay doubles, up to MaxRetryDelay.
//
// It must be positive. Zero does not mean "retry once immediately" -- it
// means every retry fires immediately, which turns a failing remote
// source into a tight request loop, so New replaces a non-positive value
// with the default.
RetryDelay time.Duration
// MaxRetryDelay caps the backoff between retries (default: 30s).
//
// It must be positive for the same reason, and more sharply: http-kit's
// CalculateRetryDelay clamps every computed delay to this ceiling
// unconditionally, so a zero here removes the backoff however large
// RetryDelay is. New replaces a non-positive value with the default.
MaxRetryDelay time.Duration
}
RetryPolicy is the retry behaviour for one remote source.
In v1 these were fields on LoadOptions, which meant every source in a loader shared one policy whether or not any of them were remote.
Example ¶
Retries back off. A non-positive field is replaced by its default rather than honoured, because zero means "retry immediately, forever" -- see RetryPolicy.
package main
import (
"fmt"
"github.com/soulteary/parser-kit/v2/remotesource"
)
func main() {
f, err := remotesource.New("https://config.internal/rules.json",
remotesource.WithRetry(remotesource.RetryPolicy{MaxRetries: 5}),
)
if err != nil {
panic(err)
}
p := f.Retry()
fmt.Println(p.MaxRetries, p.RetryDelay, p.MaxRetryDelay)
}
Output: 5 1s 30s
func DefaultRetryPolicy ¶
func DefaultRetryPolicy() RetryPolicy
DefaultRetryPolicy returns the retry policy used when WithRetry is not.